[==========] Running 20 tests from 2 test cases. [----------] Global test environment set-up. [----------] 14 tests from AdminCliTest [ RUN ] AdminCliTest.TestChangeConfig I0517 06:43:54.414698 4480 test_util.cc:200] Using random seed: 683233819 I0517 06:43:54.556370 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:43:54.556514 4480 ts_itest-base.cc:114] -------------- I0517 06:43:54.556591 4480 ts_itest-base.cc:115] 3 tablet servers I0517 06:43:54.556675 4480 ts_itest-base.cc:116] 2 replicas per TS I0517 06:43:54.556751 4480 ts_itest-base.cc:117] -------------- I0517 06:43:54.559981 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:58766 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_wait_for_new_tablets_to_elect_leader=false --allow_unsafe_replication_factor=true --catalog_manager_evict_excess_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:43:54.669711 4492 flags.cc:406] Enabled unsafe flag: --allow_unsafe_replication_factor=true W0517 06:43:54.670061 4492 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:43:54.670234 4492 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:43:54.679127 4492 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:43:54.679244 4492 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:43:54.679339 4492 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:43:54.679447 4492 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:43:54.693130 4492 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --allow_unsafe_replication_factor=true --catalog_manager_evict_excess_replicas=false --catalog_manager_wait_for_new_tablets_to_elect_leader=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:58766 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:43:54.700845 4492 master_main.cc:80] Initializing master server... I0517 06:43:54.702280 4492 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 116319us I0517 06:43:54.702997 4492 fs_manager.cc:260] Metadata directory not provided I0517 06:43:54.703114 4492 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:43:54.703305 4492 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:43:54.703404 4492 server_base.cc:433] Attempting to create new FS layout instead I0517 06:43:54.710558 4492 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "8e982a9b5e224d82b0765bcdf6c74469" format_stamp: "Formatted at 2018-05-17 06:43:54 on dist-test-slave-dist-test-slave-26mv" I0517 06:43:54.711200 4492 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "8e982a9b5e224d82b0765bcdf6c74469" format_stamp: "Formatted at 2018-05-17 06:43:54 on dist-test-slave-dist-test-slave-26mv" I0517 06:43:54.720396 4492 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:43:54.721473 4492 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:43:54.721885 4492 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:43:54.728232 4492 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:43:54.728338 4492 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "8e982a9b5e224d82b0765bcdf6c74469" format_stamp: "Formatted at 2018-05-17 06:43:54 on dist-test-slave-dist-test-slave-26mv" I0517 06:43:54.728528 4492 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:43:54.882669 4492 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:43:54.884295 4492 master_main.cc:83] Starting Master server... I0517 06:43:54.922910 4492 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:58766 I0517 06:43:54.923748 4492 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:43:54.923848 4492 webserver.cc:184] Document root disabled I0517 06:43:54.925056 4492 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:45870/ I0517 06:43:54.925750 4492 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:43:54.928289 4553 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:43:54.929461 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 4492 I0517 06:43:54.930408 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:58766 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0517 06:43:54.941669 4553 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469: Bootstrap starting. I0517 06:43:54.943954 4553 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469: No blocks or log segments found. Creating new log. I0517 06:43:54.945129 4553 log.cc:520] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469: Log is configured to *not* fsync() on all Append() calls I0517 06:43:54.947201 4553 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469: No bootstrap required, opened a new log I0517 06:43:54.958638 4553 raft_consensus.cc:303] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e982a9b5e224d82b0765bcdf6c74469" member_type: VOTER } I0517 06:43:54.958943 4553 raft_consensus.cc:329] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:43:54.959065 4553 raft_consensus.cc:604] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e982a9b5e224d82b0765bcdf6c74469, State: Initialized, Role: FOLLOWER I0517 06:43:54.959470 4553 consensus_queue.cc:227] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e982a9b5e224d82b0765bcdf6c74469" member_type: VOTER } I0517 06:43:54.959776 4553 raft_consensus.cc:343] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:43:54.959918 4553 raft_consensus.cc:435] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:43:54.960057 4553 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469: Snoozing failure detection for 1.921s (starting election) I0517 06:43:54.960191 4553 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:43:54.962532 4553 raft_consensus.cc:457] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e982a9b5e224d82b0765bcdf6c74469" member_type: VOTER } I0517 06:43:54.962971 4553 leader_election.cc:270] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:43:54.964308 4556 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469: Snoozing failure detection for 1.632s (election complete) I0517 06:43:54.964527 4556 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:43:54.970261 4556 raft_consensus.cc:571] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [term 1 LEADER]: Becoming Leader. State: Replica: 8e982a9b5e224d82b0765bcdf6c74469, State: Running, Role: LEADER I0517 06:43:54.970569 4556 consensus_queue.cc:204] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e982a9b5e224d82b0765bcdf6c74469" member_type: VOTER } I0517 06:43:54.983350 4558 sys_catalog.cc:335] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e982a9b5e224d82b0765bcdf6c74469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e982a9b5e224d82b0765bcdf6c74469" member_type: VOTER } } I0517 06:43:54.983835 4558 sys_catalog.cc:338] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [sys.catalog]: This master's current role is: LEADER WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:43:55.030395 4554 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0517 06:43:55.030761 4554 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:43:55.030966 4554 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:43:55.038864 4554 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:43:55.039201 4554 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:43:54.983712 4557 sys_catalog.cc:335] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e982a9b5e224d82b0765bcdf6c74469. Latest consensus state: current_term: 1 leader_uuid: "8e982a9b5e224d82b0765bcdf6c74469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e982a9b5e224d82b0765bcdf6c74469" member_type: VOTER } } I0517 06:43:55.041354 4557 sys_catalog.cc:338] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [sys.catalog]: This master's current role is: LEADER I0517 06:43:54.983481 4553 sys_catalog.cc:335] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "8e982a9b5e224d82b0765bcdf6c74469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e982a9b5e224d82b0765bcdf6c74469" member_type: VOTER } } I0517 06:43:55.044692 4553 sys_catalog.cc:338] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [sys.catalog]: This master's current role is: LEADER I0517 06:43:55.045531 4553 sys_catalog.cc:422] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:43:55.045732 4560 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:43:55.046959 4492 master_main.cc:86] Master server successfully started. I0517 06:43:55.050086 4560 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:43:55.051677 4554 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:58766 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:43:55.059851 4554 tablet_server_main.cc:85] Initializing tablet server... I0517 06:43:55.061697 4554 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 116819us I0517 06:43:55.062175 4554 fs_manager.cc:260] Metadata directory not provided I0517 06:43:55.062301 4554 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:43:55.062535 4554 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:43:55.062661 4554 server_base.cc:433] Attempting to create new FS layout instead I0517 06:43:55.070284 4554 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "2e4229deed6345d49e7fd2155ce9fda2" format_stamp: "Formatted at 2018-05-17 06:43:55 on dist-test-slave-dist-test-slave-26mv" I0517 06:43:55.070972 4554 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "2e4229deed6345d49e7fd2155ce9fda2" format_stamp: "Formatted at 2018-05-17 06:43:55 on dist-test-slave-dist-test-slave-26mv" I0517 06:43:55.080266 4554 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:43:55.081446 4554 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:43:55.081857 4554 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:43:55.088212 4554 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:43:55.088344 4554 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "2e4229deed6345d49e7fd2155ce9fda2" format_stamp: "Formatted at 2018-05-17 06:43:55 on dist-test-slave-dist-test-slave-26mv" I0517 06:43:55.088557 4554 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:43:55.110038 4560 catalog_manager.cc:899] Generated new certificate authority record I0517 06:43:55.111529 4560 catalog_manager.cc:1021] Loading token signing keys... I0517 06:43:55.124500 4560 catalog_manager.cc:3962] T 00000000000000000000000000000000 P 8e982a9b5e224d82b0765bcdf6c74469: Generated new TSK 0 I0517 06:43:55.269459 4554 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:43:55.271805 4554 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:43:55.272755 4554 tablet_server_main.cc:90] Starting tablet server... I0517 06:43:55.376265 4554 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:44572 I0517 06:43:55.377151 4554 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:43:55.377246 4554 webserver.cc:184] Document root disabled I0517 06:43:55.378453 4554 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:44748/ I0517 06:43:55.379143 4554 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:43:55.381429 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 4554 I0517 06:43:55.381525 4554 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:43:55.382277 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:58766 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0517 06:43:55.411232 4685 heartbeater.cc:346] Connected to a master server at 127.4.96.62:58766 I0517 06:43:55.411455 4685 heartbeater.cc:426] Registering TS with master... I0517 06:43:55.411862 4685 heartbeater.cc:475] Master 127.4.96.62:58766 requested a full tablet report, sending... I0517 06:43:55.413424 4520 ts_manager.cc:81] Registered new tserver with Master: 2e4229deed6345d49e7fd2155ce9fda2 (127.4.96.1:44572) I0517 06:43:55.415076 4520 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:38375 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:43:55.491159 4688 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0517 06:43:55.491478 4688 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:43:55.491662 4688 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:43:55.499505 4688 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:43:55.499821 4688 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:43:55.522930 4688 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:58766 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:43:55.530959 4688 tablet_server_main.cc:85] Initializing tablet server... I0517 06:43:55.532698 4688 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 116819us I0517 06:43:55.533126 4688 fs_manager.cc:260] Metadata directory not provided I0517 06:43:55.533236 4688 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:43:55.533416 4688 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:43:55.533519 4688 server_base.cc:433] Attempting to create new FS layout instead I0517 06:43:55.540979 4688 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" format_stamp: "Formatted at 2018-05-17 06:43:55 on dist-test-slave-dist-test-slave-26mv" I0517 06:43:55.541612 4688 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" format_stamp: "Formatted at 2018-05-17 06:43:55 on dist-test-slave-dist-test-slave-26mv" I0517 06:43:55.559037 4688 fs_manager.cc:495] Time spent creating directory manager: real 0.017s user 0.008s sys 0.012s I0517 06:43:55.560102 4688 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:43:55.560477 4688 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:43:55.566773 4688 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.004s I0517 06:43:55.566879 4688 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" format_stamp: "Formatted at 2018-05-17 06:43:55 on dist-test-slave-dist-test-slave-26mv" I0517 06:43:55.567064 4688 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:43:55.760908 4688 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:43:55.763263 4688 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:43:55.764154 4688 tablet_server_main.cc:90] Starting tablet server... I0517 06:43:55.962357 4688 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:37860 I0517 06:43:55.963356 4688 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:43:55.963486 4688 webserver.cc:184] Document root disabled I0517 06:43:55.974328 4688 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:52189/ I0517 06:43:55.975018 4688 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:43:55.978703 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 4688 I0517 06:43:55.979552 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:58766 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0517 06:43:55.997962 4688 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:43:56.022611 4810 heartbeater.cc:346] Connected to a master server at 127.4.96.62:58766 I0517 06:43:56.022830 4810 heartbeater.cc:426] Registering TS with master... I0517 06:43:56.023221 4810 heartbeater.cc:475] Master 127.4.96.62:58766 requested a full tablet report, sending... I0517 06:43:56.024535 4520 ts_manager.cc:81] Registered new tserver with Master: bc53d8f59e0f4eb6a9168d7162d16e90 (127.4.96.2:37860) I0517 06:43:56.025609 4520 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:55215 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:43:56.081531 4811 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0517 06:43:56.081852 4811 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:43:56.082047 4811 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:43:56.089835 4811 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:43:56.090159 4811 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:43:56.107740 4811 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:58766 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:43:56.115840 4811 tablet_server_main.cc:85] Initializing tablet server... I0517 06:43:56.117604 4811 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 117319us I0517 06:43:56.118027 4811 fs_manager.cc:260] Metadata directory not provided I0517 06:43:56.118137 4811 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:43:56.118314 4811 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:43:56.118418 4811 server_base.cc:433] Attempting to create new FS layout instead I0517 06:43:56.125939 4811 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "8e4eb3f039b94bdf8696662c2647b757" format_stamp: "Formatted at 2018-05-17 06:43:56 on dist-test-slave-dist-test-slave-26mv" I0517 06:43:56.126581 4811 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "8e4eb3f039b94bdf8696662c2647b757" format_stamp: "Formatted at 2018-05-17 06:43:56 on dist-test-slave-dist-test-slave-26mv" I0517 06:43:56.135576 4811 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:43:56.136672 4811 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:43:56.137058 4811 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:43:56.143308 4811 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:43:56.143415 4811 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "8e4eb3f039b94bdf8696662c2647b757" format_stamp: "Formatted at 2018-05-17 06:43:56 on dist-test-slave-dist-test-slave-26mv" I0517 06:43:56.143600 4811 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:43:56.287957 4811 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:43:56.290266 4811 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:43:56.291188 4811 tablet_server_main.cc:90] Starting tablet server... I0517 06:43:56.391346 4811 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:53527 I0517 06:43:56.392205 4811 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:43:56.392298 4811 webserver.cc:184] Document root disabled I0517 06:43:56.393363 4811 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:54382/ I0517 06:43:56.394044 4811 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:43:56.397130 4811 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:43:56.403146 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 4811 I0517 06:43:56.417328 4685 heartbeater.cc:467] Master 127.4.96.62:58766 was elected leader, sending a full tablet report... I0517 06:43:56.417460 4935 heartbeater.cc:346] Connected to a master server at 127.4.96.62:58766 I0517 06:43:56.417678 4935 heartbeater.cc:426] Registering TS with master... I0517 06:43:56.418105 4935 heartbeater.cc:475] Master 127.4.96.62:58766 requested a full tablet report, sending... I0517 06:43:56.420258 4520 ts_manager.cc:81] Registered new tserver with Master: 8e4eb3f039b94bdf8696662c2647b757 (127.4.96.3:53527) I0517 06:43:56.421335 4520 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:44978 I0517 06:43:56.428509 4480 external_mini_cluster.cc:482] 3 TS(s) registered with all masters I0517 06:43:56.456172 4520 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41319: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:43:56.493099 4747 tablet_service.cc:759] Processing CreateTablet for tablet 4d7fabdb13744d43bf3acde7ffa68834 (table=TestTable [id=d57d87d07ed94a769cbfec5a0bd4c4e9]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:43:56.494371 4747 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4d7fabdb13744d43bf3acde7ffa68834. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:43:56.494989 4872 tablet_service.cc:759] Processing CreateTablet for tablet 4d7fabdb13744d43bf3acde7ffa68834 (table=TestTable [id=d57d87d07ed94a769cbfec5a0bd4c4e9]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:43:56.495585 4872 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4d7fabdb13744d43bf3acde7ffa68834. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:43:56.507021 4747 ts_tablet_manager.cc:1173] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90: Registered tablet (data state: TABLET_DATA_READY) I0517 06:43:56.507905 4872 ts_tablet_manager.cc:1173] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757: Registered tablet (data state: TABLET_DATA_READY) I0517 06:43:56.509503 4953 ts_tablet_manager.cc:1028] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90: Bootstrapping tablet I0517 06:43:56.509898 4953 tablet_bootstrap.cc:436] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90: Bootstrap starting. I0517 06:43:56.512579 4953 tablet_bootstrap.cc:587] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90: No blocks or log segments found. Creating new log. I0517 06:43:56.513638 4953 log.cc:520] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90: Log is configured to *not* fsync() on all Append() calls I0517 06:43:56.516008 4954 ts_tablet_manager.cc:1028] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757: Bootstrapping tablet I0517 06:43:56.516338 4954 tablet_bootstrap.cc:436] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757: Bootstrap starting. I0517 06:43:56.516953 4953 tablet_bootstrap.cc:436] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90: No bootstrap required, opened a new log I0517 06:43:56.517271 4953 ts_tablet_manager.cc:1045] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s I0517 06:43:56.519845 4954 tablet_bootstrap.cc:587] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757: No blocks or log segments found. Creating new log. I0517 06:43:56.520969 4954 log.cc:520] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757: Log is configured to *not* fsync() on all Append() calls I0517 06:43:56.523780 4954 tablet_bootstrap.cc:436] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757: No bootstrap required, opened a new log I0517 06:43:56.524034 4954 ts_tablet_manager.cc:1045] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s I0517 06:43:56.529335 4953 raft_consensus.cc:303] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } I0517 06:43:56.529809 4953 raft_consensus.cc:604] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc53d8f59e0f4eb6a9168d7162d16e90, State: Initialized, Role: FOLLOWER I0517 06:43:56.530082 4953 consensus_queue.cc:227] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } I0517 06:43:56.531349 4810 heartbeater.cc:467] Master 127.4.96.62:58766 was elected leader, sending a full tablet report... I0517 06:43:56.532716 4953 ts_tablet_manager.cc:1073] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s I0517 06:43:56.535799 4954 raft_consensus.cc:303] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } I0517 06:43:56.536321 4954 raft_consensus.cc:604] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4eb3f039b94bdf8696662c2647b757, State: Initialized, Role: FOLLOWER I0517 06:43:56.537395 4954 consensus_queue.cc:227] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } I0517 06:43:56.541285 4935 heartbeater.cc:467] Master 127.4.96.62:58766 was elected leader, sending a full tablet report... I0517 06:43:56.543051 4480 external_mini_cluster.cc:482] 3 TS(s) registered with all masters I0517 06:43:56.545199 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2e4229deed6345d49e7fd2155ce9fda2 to finish bootstrapping I0517 06:43:56.558212 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bc53d8f59e0f4eb6a9168d7162d16e90 to finish bootstrapping I0517 06:43:56.566881 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8e4eb3f039b94bdf8696662c2647b757 to finish bootstrapping I0517 06:43:56.541574 4954 ts_tablet_manager.cc:1073] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757: Time spent starting tablet: real 0.017s user 0.016s sys 0.004s I0517 06:43:56.585755 4892 tablet_service.cc:1110] Received Run Leader Election RPC: tablet_id: "4d7fabdb13744d43bf3acde7ffa68834" dest_uuid: "8e4eb3f039b94bdf8696662c2647b757" from {username='slave'} at 127.0.0.1:46030 I0517 06:43:56.586086 4892 raft_consensus.cc:435] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [term 0 FOLLOWER]: Starting forced leader election (received explicit request) I0517 06:43:56.586254 4892 raft_consensus.cc:2756] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:43:56.589556 4892 raft_consensus.cc:457] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } I0517 06:43:56.590286 4892 leader_election.cc:243] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [CANDIDATE]: Term 1 election: Requesting vote from peer bc53d8f59e0f4eb6a9168d7162d16e90 (127.4.96.2:37860) I0517 06:43:56.600394 4767 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4d7fabdb13744d43bf3acde7ffa68834" candidate_uuid: "8e4eb3f039b94bdf8696662c2647b757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" I0517 06:43:56.600827 4767 raft_consensus.cc:2756] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:43:56.604385 4767 raft_consensus.cc:2219] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e4eb3f039b94bdf8696662c2647b757 in term 1. I0517 06:43:56.605263 4824 leader_election.cc:379] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [CANDIDATE]: Term 1 election: Vote granted by peer bc53d8f59e0f4eb6a9168d7162d16e90 (127.4.96.2:37860) I0517 06:43:56.605429 4824 leader_election.cc:270] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:43:56.606552 4958 raft_consensus.cc:2511] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:43:56.607527 4958 raft_consensus.cc:571] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [term 1 LEADER]: Becoming Leader. State: Replica: 8e4eb3f039b94bdf8696662c2647b757, State: Running, Role: LEADER I0517 06:43:56.607906 4958 consensus_queue.cc:204] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } I0517 06:43:56.612300 4480 cluster_itest_util.cc:248] Not converged past 1 yet: 0.0 1.1 I0517 06:43:56.612932 4519 catalog_manager.cc:3686] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 reported cstate change: term changed from 0 to 1, leader changed from to 8e4eb3f039b94bdf8696662c2647b757 (127.4.96.3). New cstate: current_term: 1 leader_uuid: "8e4eb3f039b94bdf8696662c2647b757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } health_report { overall_health: UNKNOWN } } } W0517 06:43:56.647483 4936 tablet.cc:1678] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:43:56.715042 4480 cluster_itest_util.cc:248] Not converged past 1 yet: 0.0 1.1 W0517 06:43:56.745496 4812 tablet.cc:1678] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:43:56.918365 4480 cluster_itest_util.cc:248] Not converged past 1 yet: 0.0 1.1 I0517 06:43:57.083659 4766 raft_consensus.cc:1057] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90 [term 1 FOLLOWER]: Refusing update from remote peer 8e4eb3f039b94bdf8696662c2647b757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:43:57.085103 4958 consensus_queue.cc:963] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:43:57.221514 4480 test_util.cc:200] Using random seed: 686040691 I0517 06:43:57.241729 4480 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0517 06:43:57.247807 4480 kudu-admin-test.cc:167] Adding replica at tserver with UUID 2e4229deed6345d49e7fd2155ce9fda2 as VOTER... W0517 06:43:57.414907 4973 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:43:57.730604 4892 consensus_peers.cc:456] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 -> Peer bc53d8f59e0f4eb6a9168d7162d16e90 (127.4.96.2:37860): Closing peer: bc53d8f59e0f4eb6a9168d7162d16e90 I0517 06:43:57.731104 4892 consensus_queue.cc:204] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.49, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } peers { permanent_uuid: "2e4229deed6345d49e7fd2155ce9fda2" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 44572 } } I0517 06:43:57.735263 4766 raft_consensus.cc:1057] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90 [term 1 FOLLOWER]: Refusing update from remote peer 8e4eb3f039b94bdf8696662c2647b757: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 51. (index mismatch) I0517 06:43:57.736244 4962 consensus_queue.cc:963] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s I0517 06:43:57.741837 4958 raft_consensus.cc:2658] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [term 1 LEADER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER 2e4229deed6345d49e7fd2155ce9fda2 (127.4.96.1) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } peers { permanent_uuid: "2e4229deed6345d49e7fd2155ce9fda2" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 44572 } } } I0517 06:43:57.743161 4766 raft_consensus.cc:2658] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90 [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER 2e4229deed6345d49e7fd2155ce9fda2 (127.4.96.1) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } peers { permanent_uuid: "2e4229deed6345d49e7fd2155ce9fda2" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 44572 } } } I0517 06:43:57.751518 4519 catalog_manager.cc:3686] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90 reported cstate change: config changed from index -1 to 50, VOTER 2e4229deed6345d49e7fd2155ce9fda2 (127.4.96.1) added. New cstate: current_term: 1 leader_uuid: "8e4eb3f039b94bdf8696662c2647b757" committed_config { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } peers { permanent_uuid: "2e4229deed6345d49e7fd2155ce9fda2" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 44572 } } } W0517 06:43:57.773993 4824 consensus_peers.cc:434] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 -> Peer 2e4229deed6345d49e7fd2155ce9fda2 (127.4.96.1:44572): Couldn't send request to peer 2e4229deed6345d49e7fd2155ce9fda2 for tablet 4d7fabdb13744d43bf3acde7ffa68834. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d7fabdb13744d43bf3acde7ffa68834. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:43:58.154829 5001 ts_tablet_manager.cc:669] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: Initiating tablet copy from peer 8e4eb3f039b94bdf8696662c2647b757 (127.4.96.3:53527) I0517 06:43:58.155141 5001 tablet_copy_client.cc:239] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.3:53527 I0517 06:43:58.168611 4912 tablet_copy_service.cc:133] P 8e4eb3f039b94bdf8696662c2647b757: Received BeginTabletCopySession request for tablet 4d7fabdb13744d43bf3acde7ffa68834 from peer 2e4229deed6345d49e7fd2155ce9fda2 ({username='slave'} at 127.4.96.1:47145) I0517 06:43:58.168871 4912 tablet_copy_service.cc:154] P 8e4eb3f039b94bdf8696662c2647b757: Beginning new tablet copy session on tablet 4d7fabdb13744d43bf3acde7ffa68834 from peer 2e4229deed6345d49e7fd2155ce9fda2 at {username='slave'} at 127.4.96.1:47145: session id = 2e4229deed6345d49e7fd2155ce9fda2-4d7fabdb13744d43bf3acde7ffa68834 I0517 06:43:58.172823 4912 tablet_copy_source_session.cc:202] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:43:58.175400 5001 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4d7fabdb13744d43bf3acde7ffa68834. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:43:58.188721 5001 ts_tablet_manager.cc:1173] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:43:58.189085 5001 tablet_copy_client.cc:565] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: tablet copy: Starting download of 0 data blocks... I0517 06:43:58.189431 5001 tablet_copy_client.cc:528] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: tablet copy: Starting download of 1 WAL segments... I0517 06:43:58.192641 5001 tablet_copy_client.cc:404] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:43:58.196574 5001 ts_tablet_manager.cc:1028] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: Bootstrapping tablet I0517 06:43:58.196913 5001 tablet_bootstrap.cc:436] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: Bootstrap starting. I0517 06:43:58.221169 5001 log.cc:520] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: Log is configured to *not* fsync() on all Append() calls I0517 06:43:58.402740 5001 tablet_bootstrap.cc:436] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: Bootstrap replayed 1/1 log segments. Stats: ops{read=109 overwritten=0 applied=108 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0517 06:43:58.403311 5001 tablet_bootstrap.cc:436] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: Bootstrap complete. I0517 06:43:58.403690 5001 ts_tablet_manager.cc:1045] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: Time spent bootstrapping tablet: real 0.207s user 0.164s sys 0.024s I0517 06:43:58.411931 5001 raft_consensus.cc:303] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } peers { permanent_uuid: "2e4229deed6345d49e7fd2155ce9fda2" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 44572 } } I0517 06:43:58.415783 5001 raft_consensus.cc:604] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4229deed6345d49e7fd2155ce9fda2, State: Initialized, Role: FOLLOWER I0517 06:43:58.416188 5001 consensus_queue.cc:227] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 108, Last appended: 1.109, Last appended by leader: 109, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } peers { permanent_uuid: "2e4229deed6345d49e7fd2155ce9fda2" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 44572 } } I0517 06:43:58.423429 5001 ts_tablet_manager.cc:1073] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: Time spent starting tablet: real 0.020s user 0.012s sys 0.004s I0517 06:43:58.424916 4912 tablet_copy_service.cc:323] P 8e4eb3f039b94bdf8696662c2647b757: Request end of tablet copy session 2e4229deed6345d49e7fd2155ce9fda2-4d7fabdb13744d43bf3acde7ffa68834 received from {username='slave'} at 127.4.96.1:47145 I0517 06:43:58.425153 4912 tablet_copy_service.cc:411] P 8e4eb3f039b94bdf8696662c2647b757: Ending tablet copy session 2e4229deed6345d49e7fd2155ce9fda2-4d7fabdb13744d43bf3acde7ffa68834 on tablet 4d7fabdb13744d43bf3acde7ffa68834 with peer 2e4229deed6345d49e7fd2155ce9fda2 I0517 06:43:58.592051 4642 raft_consensus.cc:1002] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.49->[1.50-1.162] Dedup: 1.109->[1.110-1.162] I0517 06:43:58.778213 4480 kudu-admin-test.cc:187] Waiting for replicas to agree... I0517 06:43:58.793802 4480 kudu-admin-test.cc:198] Number of rows inserted: 182 Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 8e982a9b5e224d82b0765bcdf6c74469 | 127.4.96.62:58766 | HEALTHY Tablet Server Summary UUID | Address | Status ----------------------------------+------------------+--------- 2e4229deed6345d49e7fd2155ce9fda2 | 127.4.96.1:44572 | HEALTHY 8e4eb3f039b94bdf8696662c2647b757 | 127.4.96.3:53527 | HEALTHY bc53d8f59e0f4eb6a9168d7162d16e90 | 127.4.96.2:37860 | HEALTHY Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 OK I0517 06:43:58.879758 4480 log_verifier.cc:121] Checking tablet 4d7fabdb13744d43bf3acde7ffa68834 I0517 06:43:59.096623 4480 log_verifier.cc:172] Verified matching terms for 184 ops in tablet 4d7fabdb13744d43bf3acde7ffa68834 I0517 06:43:59.134057 4480 kudu-admin-test.cc:205] Removing replica at tserver with UUID 2e4229deed6345d49e7fd2155ce9fda2 from the config... W0517 06:43:59.259461 5042 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:43:59.639753 4892 consensus_peers.cc:456] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 -> Peer 2e4229deed6345d49e7fd2155ce9fda2 (127.4.96.1:44572): Closing peer: 2e4229deed6345d49e7fd2155ce9fda2 I0517 06:43:59.639937 4892 consensus_peers.cc:456] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 -> Peer bc53d8f59e0f4eb6a9168d7162d16e90 (127.4.96.2:37860): Closing peer: bc53d8f59e0f4eb6a9168d7162d16e90 I0517 06:43:59.640374 4892 consensus_queue.cc:204] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 184, Committed index: 184, Last appended: 1.184, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } I0517 06:43:59.643245 4767 raft_consensus.cc:1057] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90 [term 1 FOLLOWER]: Refusing update from remote peer 8e4eb3f039b94bdf8696662c2647b757: Log matching property violated. Preceding OpId in replica: term: 1 index: 184. Preceding OpId from leader: term: 1 index: 185. (index mismatch) I0517 06:43:59.644337 4958 consensus_queue.cc:963] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 185, Last known committed idx: 184, Time since last communication: 0.000s I0517 06:43:59.658450 5056 raft_consensus.cc:2658] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 [term 1 LEADER]: Committing config change with OpId 1.185: config changed from index 50 to 185, VOTER 2e4229deed6345d49e7fd2155ce9fda2 (127.4.96.1) evicted. New config: { opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } } I0517 06:43:59.659209 4767 raft_consensus.cc:2658] T 4d7fabdb13744d43bf3acde7ffa68834 P bc53d8f59e0f4eb6a9168d7162d16e90 [term 1 FOLLOWER]: Committing config change with OpId 1.185: config changed from index 50 to 185, VOTER 2e4229deed6345d49e7fd2155ce9fda2 (127.4.96.1) evicted. New config: { opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } } } I0517 06:43:59.669385 4520 catalog_manager.cc:3686] T 4d7fabdb13744d43bf3acde7ffa68834 P 8e4eb3f039b94bdf8696662c2647b757 reported cstate change: config changed from index 50 to 185, VOTER 2e4229deed6345d49e7fd2155ce9fda2 (127.4.96.1) evicted. New cstate: current_term: 1 leader_uuid: "8e4eb3f039b94bdf8696662c2647b757" committed_config { opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "8e4eb3f039b94bdf8696662c2647b757" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc53d8f59e0f4eb6a9168d7162d16e90" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37860 } health_report { overall_health: HEALTHY } } } I0517 06:43:59.697077 4520 catalog_manager.cc:3013] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 4d7fabdb13744d43bf3acde7ffa68834 on 2e4229deed6345d49e7fd2155ce9fda2 (127.4.96.1:44572) (TS 2e4229deed6345d49e7fd2155ce9fda2 not found in new config with opid_index 185) I0517 06:43:59.718417 4622 tablet_service.cc:800] Processing DeleteTablet for tablet 4d7fabdb13744d43bf3acde7ffa68834 with delete_type TABLET_DATA_TOMBSTONED (TS 2e4229deed6345d49e7fd2155ce9fda2 not found in new config with opid_index 185) from {username='slave'} at 127.0.0.1:55008 I0517 06:43:59.730648 5065 tablet_replica.cc:261] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: stopping tablet replica I0517 06:43:59.730893 5065 maintenance_manager.cc:237] P 2e4229deed6345d49e7fd2155ce9fda2: Unregistered op CompactRowSetsOp(4d7fabdb13744d43bf3acde7ffa68834) I0517 06:43:59.731065 5065 maintenance_manager.cc:237] P 2e4229deed6345d49e7fd2155ce9fda2: Unregistered op MinorDeltaCompactionOp(4d7fabdb13744d43bf3acde7ffa68834) I0517 06:43:59.731225 5065 maintenance_manager.cc:237] P 2e4229deed6345d49e7fd2155ce9fda2: Unregistered op MajorDeltaCompactionOp(4d7fabdb13744d43bf3acde7ffa68834) I0517 06:43:59.731367 5065 maintenance_manager.cc:237] P 2e4229deed6345d49e7fd2155ce9fda2: Unregistered op UndoDeltaBlockGCOp(4d7fabdb13744d43bf3acde7ffa68834) I0517 06:43:59.731542 5065 maintenance_manager.cc:237] P 2e4229deed6345d49e7fd2155ce9fda2: Unregistered op FlushMRSOp(4d7fabdb13744d43bf3acde7ffa68834) I0517 06:43:59.731683 5065 maintenance_manager.cc:237] P 2e4229deed6345d49e7fd2155ce9fda2: Unregistered op FlushDeltaMemStoresOp(4d7fabdb13744d43bf3acde7ffa68834) I0517 06:43:59.731812 5065 maintenance_manager.cc:237] P 2e4229deed6345d49e7fd2155ce9fda2: Unregistered op LogGCOp(4d7fabdb13744d43bf3acde7ffa68834) I0517 06:43:59.731990 5065 raft_consensus.cc:2016] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2 [term 1 FOLLOWER]: Raft consensus shutting down. I0517 06:43:59.732264 5065 raft_consensus.cc:2043] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2 [term 1 FOLLOWER]: Raft consensus is shut down! I0517 06:43:59.735107 5065 ts_tablet_manager.cc:1369] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0517 06:43:59.740716 5065 ts_tablet_manager.cc:1382] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.184 I0517 06:43:59.740936 5065 log.cc:981] T 4d7fabdb13744d43bf3acde7ffa68834 P 2e4229deed6345d49e7fd2155ce9fda2: Deleting WAL directory at /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/wals/4d7fabdb13744d43bf3acde7ffa68834 I0517 06:43:59.742025 4502 catalog_manager.cc:2991] TS 2e4229deed6345d49e7fd2155ce9fda2 (127.4.96.1:44572): tablet 4d7fabdb13744d43bf3acde7ffa68834 (table TestTable [id=d57d87d07ed94a769cbfec5a0bd4c4e9]) successfully deleted I0517 06:44:00.681856 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 4554 W0517 06:44:00.704752 4500 connection.cc:512] server connection from 127.4.96.1:38375 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:38375): Connection reset by peer (error 104) W0517 06:44:00.705519 4824 connection.cc:512] client connection to 127.4.96.1:44572 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:44572): Connection reset by peer (error 104) W0517 06:44:00.707037 4826 connection.cc:512] server connection from 127.4.96.1:47145 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:47145): Connection reset by peer (error 104) W0517 06:44:00.707890 4502 connection.cc:512] client connection to 127.4.96.1:44572 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:44572): Connection reset by peer (error 104) I0517 06:44:00.708951 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 4688 W0517 06:44:00.704900 4491 connection.cc:512] client connection to 127.4.96.1:44572 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:44572): Connection reset by peer (error 104) W0517 06:44:00.706593 4488 connection.cc:512] client connection to 127.4.96.1:44572 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:44572): Connection reset by peer (error 104) W0517 06:44:00.738479 4502 connection.cc:512] client connection to 127.4.96.2:37860 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:37860): Connection reset by peer (error 104) W0517 06:44:00.738528 4500 connection.cc:512] server connection from 127.4.96.2:55215 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:55215): Connection reset by peer (error 104) W0517 06:44:00.739233 4824 connection.cc:512] client connection to 127.4.96.2:37860 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:37860): Connection reset by peer (error 104) W0517 06:44:00.739224 4491 connection.cc:512] client connection to 127.4.96.2:37860 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:37860): Connection reset by peer (error 104) I0517 06:44:00.739367 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 4811 W0517 06:44:00.740391 4488 connection.cc:512] client connection to 127.4.96.2:37860 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:37860): Connection reset by peer (error 104) W0517 06:44:00.780040 4504 connection.cc:512] server connection from 127.4.96.3:44978 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:44978): Connection reset by peer (error 104) W0517 06:44:00.780117 4490 connection.cc:512] client connection to 127.4.96.3:53527 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:53527): Connection reset by peer (error 104) W0517 06:44:00.780617 4504 connection.cc:512] client connection to 127.4.96.3:53527 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:53527): Connection reset by peer (error 104) I0517 06:44:00.781141 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 4492 W0517 06:44:00.781363 4491 connection.cc:512] client connection to 127.4.96.3:53527 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:53527): Connection reset by peer (error 104) W0517 06:44:00.803704 4491 connection.cc:512] client connection to 127.4.96.62:58766 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:58766): Connection reset by peer (error 104) [ OK ] AdminCliTest.TestChangeConfig (6402 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigOnSingleFollower I0517 06:44:00.815937 4480 test_util.cc:200] Using random seed: 689635112 I0517 06:44:00.819034 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:44:00.819136 4480 ts_itest-base.cc:114] -------------- I0517 06:44:00.819226 4480 ts_itest-base.cc:115] 5 tablet servers I0517 06:44:00.819327 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:44:00.819419 4480 ts_itest-base.cc:117] -------------- I0517 06:44:00.821734 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:49034 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:00.933727 5072 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:00.934062 5072 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:00.942894 5072 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:44:00.943014 5072 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:00.943114 5072 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:44:00.943223 5072 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:44:00.957026 5072 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:49034 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:00.964711 5072 master_main.cc:80] Initializing master server... I0517 06:44:00.966208 5072 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 119319us I0517 06:44:00.966919 5072 fs_manager.cc:260] Metadata directory not provided I0517 06:44:00.967044 5072 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:00.967247 5072 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:44:00.967355 5072 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:00.974674 5072 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "d199232303f34e778806ea6eeafe02cb" format_stamp: "Formatted at 2018-05-17 06:44:00 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:00.975325 5072 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "d199232303f34e778806ea6eeafe02cb" format_stamp: "Formatted at 2018-05-17 06:44:00 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:00.984810 5072 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.012s sys 0.000s I0517 06:44:00.985910 5072 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:00.986330 5072 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:00.992651 5072 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:44:00.992758 5072 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "d199232303f34e778806ea6eeafe02cb" format_stamp: "Formatted at 2018-05-17 06:44:00 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:00.992939 5072 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:01.147557 5072 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:01.149237 5072 master_main.cc:83] Starting Master server... I0517 06:44:01.188180 5072 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:49034 I0517 06:44:01.189011 5072 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:44:01.189116 5072 webserver.cc:184] Document root disabled I0517 06:44:01.190305 5072 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:56845/ I0517 06:44:01.191005 5072 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:44:01.193511 5133 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:01.194382 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 5072 I0517 06:44:01.195677 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:49034 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:01.209506 5133 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: Bootstrap starting. I0517 06:44:01.211699 5133 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: No blocks or log segments found. Creating new log. I0517 06:44:01.212824 5133 log.cc:520] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: Log is configured to *not* fsync() on all Append() calls I0517 06:44:01.215152 5133 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: No bootstrap required, opened a new log I0517 06:44:01.227530 5133 raft_consensus.cc:303] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } I0517 06:44:01.227830 5133 raft_consensus.cc:329] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:01.227957 5133 raft_consensus.cc:604] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d199232303f34e778806ea6eeafe02cb, State: Initialized, Role: FOLLOWER I0517 06:44:01.228397 5133 consensus_queue.cc:227] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } I0517 06:44:01.228842 5133 raft_consensus.cc:343] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:44:01.229022 5133 raft_consensus.cc:435] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:44:01.229212 5133 raft_consensus.cc:2704] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: Snoozing failure detection for 2.152s (starting election) I0517 06:44:01.229390 5133 raft_consensus.cc:2756] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:01.232859 5133 raft_consensus.cc:457] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } I0517 06:44:01.233414 5133 leader_election.cc:270] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:44:01.246505 5136 raft_consensus.cc:2704] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: Snoozing failure detection for 2.116s (election complete) I0517 06:44:01.246794 5136 raft_consensus.cc:2511] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:44:01.258405 5136 raft_consensus.cc:571] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 1 LEADER]: Becoming Leader. State: Replica: d199232303f34e778806ea6eeafe02cb, State: Running, Role: LEADER I0517 06:44:01.258816 5136 consensus_queue.cc:204] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } I0517 06:44:01.262357 5138 sys_catalog.cc:335] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d199232303f34e778806ea6eeafe02cb. Latest consensus state: current_term: 1 leader_uuid: "d199232303f34e778806ea6eeafe02cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } } I0517 06:44:01.262799 5138 sys_catalog.cc:338] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: This master's current role is: LEADER I0517 06:44:01.262727 5133 sys_catalog.cc:335] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "d199232303f34e778806ea6eeafe02cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } } I0517 06:44:01.321455 5133 sys_catalog.cc:338] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: This master's current role is: LEADER I0517 06:44:01.322610 5133 sys_catalog.cc:422] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: configured and running, proceeding with master startup. I0517 06:44:01.322840 5140 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:44:01.266294 5137 sys_catalog.cc:335] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d199232303f34e778806ea6eeafe02cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } } I0517 06:44:01.325808 5137 sys_catalog.cc:338] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: This master's current role is: LEADER I0517 06:44:01.330904 5072 master_main.cc:86] Master server successfully started. WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:01.332437 5134 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:01.332959 5134 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:44:01.334204 5140 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... W0517 06:44:01.347401 5134 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:01.347903 5134 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:44:01.368454 5134 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:49034 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:01.376384 5140 catalog_manager.cc:899] Generated new certificate authority record I0517 06:44:01.378814 5140 catalog_manager.cc:1021] Loading token signing keys... I0517 06:44:01.379086 5134 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:01.380903 5134 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 119819us I0517 06:44:01.381331 5134 fs_manager.cc:260] Metadata directory not provided I0517 06:44:01.381431 5134 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:01.381623 5134 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:44:01.381726 5134 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:01.389272 5134 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "db3928f6c5ea43bab4f27c1999b849db" format_stamp: "Formatted at 2018-05-17 06:44:01 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:01.389964 5134 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "db3928f6c5ea43bab4f27c1999b849db" format_stamp: "Formatted at 2018-05-17 06:44:01 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:01.396534 5140 catalog_manager.cc:3962] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: Generated new TSK 0 I0517 06:44:01.401003 5134 fs_manager.cc:495] Time spent creating directory manager: real 0.011s user 0.004s sys 0.004s I0517 06:44:01.402216 5134 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:01.402639 5134 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:01.409240 5134 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:44:01.409351 5134 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "db3928f6c5ea43bab4f27c1999b849db" format_stamp: "Formatted at 2018-05-17 06:44:01 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:01.409565 5134 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:01.596431 5134 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:01.599683 5134 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:01.600806 5134 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:01.755137 5134 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:53133 I0517 06:44:01.756469 5134 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:44:01.756613 5134 webserver.cc:184] Document root disabled I0517 06:44:01.758124 5134 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:53274/ I0517 06:44:01.759027 5134 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:44:01.762323 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 5134 I0517 06:44:01.763492 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:49034 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:01.770838 5134 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:01.790962 5265 heartbeater.cc:346] Connected to a master server at 127.4.96.62:49034 I0517 06:44:01.791206 5265 heartbeater.cc:426] Registering TS with master... I0517 06:44:01.791637 5265 heartbeater.cc:475] Master 127.4.96.62:49034 requested a full tablet report, sending... I0517 06:44:01.793325 5100 ts_manager.cc:81] Registered new tserver with Master: db3928f6c5ea43bab4f27c1999b849db (127.4.96.1:53133) I0517 06:44:01.795840 5100 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:42341 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:01.881731 5269 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:01.882103 5269 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:01.890102 5269 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:01.890434 5269 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:44:01.904093 5269 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:49034 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:01.912266 5269 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:01.914070 5269 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 119819us I0517 06:44:01.914502 5269 fs_manager.cc:260] Metadata directory not provided I0517 06:44:01.914643 5269 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:01.914850 5269 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:44:01.914969 5269 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:02.092036 5269 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "12bd807be9d84470a0d27627be106be4" format_stamp: "Formatted at 2018-05-17 06:44:01 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:02.093042 5269 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "12bd807be9d84470a0d27627be106be4" format_stamp: "Formatted at 2018-05-17 06:44:01 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:02.103153 5269 fs_manager.cc:495] Time spent creating directory manager: real 0.010s user 0.004s sys 0.004s I0517 06:44:02.104296 5269 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:02.104733 5269 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:02.111384 5269 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0517 06:44:02.111501 5269 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "12bd807be9d84470a0d27627be106be4" format_stamp: "Formatted at 2018-05-17 06:44:01 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:02.111724 5269 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:02.364387 5269 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:02.366945 5269 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:02.368038 5269 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:02.473603 5269 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:48598 I0517 06:44:02.474524 5269 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:44:02.474635 5269 webserver.cc:184] Document root disabled I0517 06:44:02.475965 5269 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:55046/ I0517 06:44:02.476645 5269 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:44:02.482594 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 5269 I0517 06:44:02.483721 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:49034 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:02.490512 5269 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:02.520952 5390 heartbeater.cc:346] Connected to a master server at 127.4.96.62:49034 I0517 06:44:02.521273 5390 heartbeater.cc:426] Registering TS with master... I0517 06:44:02.521865 5390 heartbeater.cc:475] Master 127.4.96.62:49034 requested a full tablet report, sending... I0517 06:44:02.523439 5100 ts_manager.cc:81] Registered new tserver with Master: 12bd807be9d84470a0d27627be106be4 (127.4.96.2:48598) I0517 06:44:02.524526 5100 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:51342 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:02.628468 5394 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:02.628816 5394 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:02.637398 5394 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:02.637727 5394 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:44:02.650858 5394 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:49034 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:02.659638 5394 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:02.661440 5394 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 120319us I0517 06:44:02.661881 5394 fs_manager.cc:260] Metadata directory not provided I0517 06:44:02.662024 5394 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:02.662241 5394 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:44:02.662353 5394 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:02.669941 5394 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "896b9662512241d29d01130917359a60" format_stamp: "Formatted at 2018-05-17 06:44:02 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:02.670626 5394 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "896b9662512241d29d01130917359a60" format_stamp: "Formatted at 2018-05-17 06:44:02 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:02.680205 5394 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:44:02.681308 5394 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:02.681713 5394 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:02.688347 5394 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:44:02.688460 5394 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "896b9662512241d29d01130917359a60" format_stamp: "Formatted at 2018-05-17 06:44:02 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:02.688678 5394 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:02.798944 5265 heartbeater.cc:467] Master 127.4.96.62:49034 was elected leader, sending a full tablet report... I0517 06:44:02.890797 5394 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:02.893172 5394 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:02.894095 5394 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:02.980159 5394 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:44651 I0517 06:44:02.981034 5394 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:44:02.981129 5394 webserver.cc:184] Document root disabled I0517 06:44:02.982339 5394 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:41440/ I0517 06:44:02.983034 5394 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:44:02.985890 5394 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:02.988112 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 5394 I0517 06:44:02.988983 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:49034 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:03.004458 5515 heartbeater.cc:346] Connected to a master server at 127.4.96.62:49034 I0517 06:44:03.004727 5515 heartbeater.cc:426] Registering TS with master... I0517 06:44:03.005210 5515 heartbeater.cc:475] Master 127.4.96.62:49034 requested a full tablet report, sending... I0517 06:44:03.006597 5100 ts_manager.cc:81] Registered new tserver with Master: 896b9662512241d29d01130917359a60 (127.4.96.3:44651) I0517 06:44:03.007725 5100 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:60327 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:03.091944 5518 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:03.092265 5518 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:03.100088 5518 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:03.100400 5518 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:44:03.112843 5518 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:49034 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:03.120790 5518 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:03.122511 5518 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 120819us I0517 06:44:03.122943 5518 fs_manager.cc:260] Metadata directory not provided I0517 06:44:03.123056 5518 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:03.123240 5518 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:44:03.123345 5518 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:03.130831 5518 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "d6e1bdcef7c443299c40504db909f7cf" format_stamp: "Formatted at 2018-05-17 06:44:03 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:03.131455 5518 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "d6e1bdcef7c443299c40504db909f7cf" format_stamp: "Formatted at 2018-05-17 06:44:03 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:03.140738 5518 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:44:03.141805 5518 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:03.142182 5518 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:03.148608 5518 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0517 06:44:03.148716 5518 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "d6e1bdcef7c443299c40504db909f7cf" format_stamp: "Formatted at 2018-05-17 06:44:03 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:03.148901 5518 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:03.357821 5518 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:03.360178 5518 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:03.361165 5518 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:03.446188 5518 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:56338 I0517 06:44:03.447082 5518 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:44:03.447190 5518 webserver.cc:184] Document root disabled I0517 06:44:03.448385 5518 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:46665/ I0517 06:44:03.449075 5518 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:44:03.453006 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 5518 I0517 06:44:03.453168 5518 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:03.454234 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.4.96.5:0 --local_ip_for_outbound_sockets=127.4.96.5 --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:49034 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:03.471210 5639 heartbeater.cc:346] Connected to a master server at 127.4.96.62:49034 I0517 06:44:03.471433 5639 heartbeater.cc:426] Registering TS with master... I0517 06:44:03.471850 5639 heartbeater.cc:475] Master 127.4.96.62:49034 requested a full tablet report, sending... I0517 06:44:03.473181 5100 ts_manager.cc:81] Registered new tserver with Master: d6e1bdcef7c443299c40504db909f7cf (127.4.96.4:56338) I0517 06:44:03.474268 5100 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:59181 I0517 06:44:03.527375 5390 heartbeater.cc:467] Master 127.4.96.62:49034 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:03.557901 5642 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:03.558238 5642 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:03.566259 5642 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:03.566587 5642 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.5 I0517 06:44:03.579149 5642 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:49034 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.5 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:03.587425 5642 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:03.589222 5642 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 120819us I0517 06:44:03.589669 5642 fs_manager.cc:260] Metadata directory not provided I0517 06:44:03.589799 5642 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:03.590031 5642 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0517 06:44:03.590150 5642 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:03.599689 5642 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "cb72eb5a50c24c5fa232a20acd802468" format_stamp: "Formatted at 2018-05-17 06:44:03 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:03.600407 5642 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "cb72eb5a50c24c5fa232a20acd802468" format_stamp: "Formatted at 2018-05-17 06:44:03 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:03.609653 5642 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:44:03.610749 5642 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:03.611124 5642 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:03.617413 5642 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:44:03.617537 5642 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal uuid: "cb72eb5a50c24c5fa232a20acd802468" format_stamp: "Formatted at 2018-05-17 06:44:03 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:03.617776 5642 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:03.813304 5642 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:03.815610 5642 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:03.816534 5642 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:03.901367 5642 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.5:37703 I0517 06:44:03.902272 5642 webserver.cc:173] Starting webserver on 127.4.96.5:0 I0517 06:44:03.902375 5642 webserver.cc:184] Document root disabled I0517 06:44:03.903553 5642 webserver.cc:311] Webserver started. Bound to: http://127.4.96.5:45302/ I0517 06:44:03.904196 5642 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb I0517 06:44:03.906703 5642 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:03.909288 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 5642 I0517 06:44:03.925170 5763 heartbeater.cc:346] Connected to a master server at 127.4.96.62:49034 I0517 06:44:03.925390 5763 heartbeater.cc:426] Registering TS with master... I0517 06:44:03.925806 5763 heartbeater.cc:475] Master 127.4.96.62:49034 requested a full tablet report, sending... I0517 06:44:03.927207 5100 ts_manager.cc:81] Registered new tserver with Master: cb72eb5a50c24c5fa232a20acd802468 (127.4.96.5:37703) I0517 06:44:03.928315 5100 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.5:51320 I0517 06:44:03.930301 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:44:03.946765 5100 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52864: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:44:03.986724 5327 tablet_service.cc:759] Processing CreateTablet for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (table=TestTable [id=8f0321e0a36d4ee2bfb4d2c2a786bbf9]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:03.987633 5327 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:03.991397 5452 tablet_service.cc:759] Processing CreateTablet for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (table=TestTable [id=8f0321e0a36d4ee2bfb4d2c2a786bbf9]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:03.992173 5452 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:04.000257 5327 ts_tablet_manager.cc:1173] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:04.002244 5576 tablet_service.cc:759] Processing CreateTablet for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (table=TestTable [id=8f0321e0a36d4ee2bfb4d2c2a786bbf9]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:04.003052 5576 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:04.003720 5452 ts_tablet_manager.cc:1173] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:04.018479 5576 ts_tablet_manager.cc:1173] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:04.020121 5783 ts_tablet_manager.cc:1028] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Bootstrapping tablet I0517 06:44:04.021767 5783 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Bootstrap starting. I0517 06:44:04.022657 5785 ts_tablet_manager.cc:1028] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Bootstrapping tablet I0517 06:44:04.024003 5785 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Bootstrap starting. I0517 06:44:04.024605 5783 tablet_bootstrap.cc:587] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: No blocks or log segments found. Creating new log. I0517 06:44:04.025776 5783 log.cc:520] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Log is configured to *not* fsync() on all Append() calls I0517 06:44:04.027458 5785 tablet_bootstrap.cc:587] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: No blocks or log segments found. Creating new log. I0517 06:44:04.028971 5785 log.cc:520] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Log is configured to *not* fsync() on all Append() calls I0517 06:44:04.036602 5785 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: No bootstrap required, opened a new log I0517 06:44:04.036954 5785 ts_tablet_manager.cc:1045] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s I0517 06:44:04.040256 5783 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: No bootstrap required, opened a new log I0517 06:44:04.040614 5783 ts_tablet_manager.cc:1045] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s I0517 06:44:04.051558 5785 raft_consensus.cc:303] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } I0517 06:44:04.052402 5785 raft_consensus.cc:329] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:04.052618 5785 raft_consensus.cc:604] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e1bdcef7c443299c40504db909f7cf, State: Initialized, Role: FOLLOWER I0517 06:44:04.053283 5785 consensus_queue.cc:227] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } I0517 06:44:04.055681 5783 raft_consensus.cc:303] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } I0517 06:44:04.056416 5783 raft_consensus.cc:329] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:04.056604 5783 raft_consensus.cc:604] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bd807be9d84470a0d27627be106be4, State: Initialized, Role: FOLLOWER I0517 06:44:04.057201 5783 consensus_queue.cc:227] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } I0517 06:44:04.059551 5639 heartbeater.cc:467] Master 127.4.96.62:49034 was elected leader, sending a full tablet report... I0517 06:44:04.061257 5788 raft_consensus.cc:435] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:44:04.062335 5788 raft_consensus.cc:2704] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Snoozing failure detection for 2.003s (starting election) I0517 06:44:04.062815 5788 raft_consensus.cc:457] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } I0517 06:44:04.009990 5515 heartbeater.cc:467] Master 127.4.96.62:49034 was elected leader, sending a full tablet report... I0517 06:44:04.065163 5784 ts_tablet_manager.cc:1028] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Bootstrapping tablet I0517 06:44:04.065634 5784 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Bootstrap starting. I0517 06:44:04.067121 5788 leader_election.cc:243] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 12bd807be9d84470a0d27627be106be4 (127.4.96.2:48598) I0517 06:44:04.067561 5788 leader_election.cc:243] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 896b9662512241d29d01130917359a60 (127.4.96.3:44651) I0517 06:44:04.068608 5784 tablet_bootstrap.cc:587] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: No blocks or log segments found. Creating new log. I0517 06:44:04.070356 5784 log.cc:520] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Log is configured to *not* fsync() on all Append() calls I0517 06:44:04.071952 5783 ts_tablet_manager.cc:1073] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Time spent starting tablet: real 0.031s user 0.016s sys 0.008s I0517 06:44:04.075759 5784 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: No bootstrap required, opened a new log I0517 06:44:04.076027 5784 ts_tablet_manager.cc:1045] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s I0517 06:44:04.089496 5784 raft_consensus.cc:303] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } I0517 06:44:04.090251 5784 raft_consensus.cc:329] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:04.090443 5784 raft_consensus.cc:604] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896b9662512241d29d01130917359a60, State: Initialized, Role: FOLLOWER I0517 06:44:04.090972 5784 consensus_queue.cc:227] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } I0517 06:44:04.099701 5784 ts_tablet_manager.cc:1073] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Time spent starting tablet: real 0.024s user 0.020s sys 0.000s I0517 06:44:04.064584 5785 ts_tablet_manager.cc:1073] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Time spent starting tablet: real 0.027s user 0.020s sys 0.000s I0517 06:44:04.142365 5347 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9e4acfa8c48f4fbebd81cd58cfd3edc7" candidate_uuid: "d6e1bdcef7c443299c40504db909f7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bd807be9d84470a0d27627be106be4" is_pre_election: true I0517 06:44:04.142483 5472 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9e4acfa8c48f4fbebd81cd58cfd3edc7" candidate_uuid: "d6e1bdcef7c443299c40504db909f7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896b9662512241d29d01130917359a60" is_pre_election: true I0517 06:44:04.142827 5347 raft_consensus.cc:2704] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Snoozing failure detection for 2.111s (vote granted) I0517 06:44:04.142890 5472 raft_consensus.cc:2704] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Snoozing failure detection for 2.061s (vote granted) I0517 06:44:04.143031 5347 raft_consensus.cc:2219] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e1bdcef7c443299c40504db909f7cf in term 0. I0517 06:44:04.143059 5472 raft_consensus.cc:2219] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e1bdcef7c443299c40504db909f7cf in term 0. I0517 06:44:04.143895 5530 leader_election.cc:379] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [CANDIDATE]: Term 1 pre-election: Vote granted by peer 896b9662512241d29d01130917359a60 (127.4.96.3:44651) I0517 06:44:04.144083 5530 leader_election.cc:270] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:44:04.144335 5527 leader_election.cc:379] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [CANDIDATE]: Term 1 pre-election: Vote granted by peer 12bd807be9d84470a0d27627be106be4 (127.4.96.2:48598) I0517 06:44:04.144508 5788 raft_consensus.cc:2704] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Snoozing failure detection for 1.825s (election complete) I0517 06:44:04.144686 5788 raft_consensus.cc:2511] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:44:04.144824 5788 raft_consensus.cc:435] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:44:04.144940 5788 raft_consensus.cc:2704] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Snoozing failure detection for 1.575s (starting election) I0517 06:44:04.145056 5788 raft_consensus.cc:2756] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:04.147816 5788 raft_consensus.cc:457] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } I0517 06:44:04.148444 5788 leader_election.cc:243] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [CANDIDATE]: Term 1 election: Requesting vote from peer 12bd807be9d84470a0d27627be106be4 (127.4.96.2:48598) I0517 06:44:04.148741 5788 leader_election.cc:243] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [CANDIDATE]: Term 1 election: Requesting vote from peer 896b9662512241d29d01130917359a60 (127.4.96.3:44651) I0517 06:44:04.149576 5347 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9e4acfa8c48f4fbebd81cd58cfd3edc7" candidate_uuid: "d6e1bdcef7c443299c40504db909f7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bd807be9d84470a0d27627be106be4" I0517 06:44:04.149699 5472 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9e4acfa8c48f4fbebd81cd58cfd3edc7" candidate_uuid: "d6e1bdcef7c443299c40504db909f7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896b9662512241d29d01130917359a60" I0517 06:44:04.149987 5472 raft_consensus.cc:2756] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:04.150014 5347 raft_consensus.cc:2756] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:04.150190 5472 raft_consensus.cc:2704] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Snoozing failure detection for 1.924s (vote granted) I0517 06:44:04.150259 5347 raft_consensus.cc:2704] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Snoozing failure detection for 2.239s (vote granted) I0517 06:44:04.152853 5472 raft_consensus.cc:2219] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e1bdcef7c443299c40504db909f7cf in term 1. I0517 06:44:04.153501 5530 leader_election.cc:379] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [CANDIDATE]: Term 1 election: Vote granted by peer 896b9662512241d29d01130917359a60 (127.4.96.3:44651) I0517 06:44:04.153672 5530 leader_election.cc:270] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:44:04.154000 5788 raft_consensus.cc:2704] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Snoozing failure detection for 2.137s (election complete) I0517 06:44:04.154176 5788 raft_consensus.cc:2511] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:44:04.154284 5347 raft_consensus.cc:2219] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e1bdcef7c443299c40504db909f7cf in term 1. I0517 06:44:04.154400 5788 raft_consensus.cc:571] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 1 LEADER]: Becoming Leader. State: Replica: d6e1bdcef7c443299c40504db909f7cf, State: Running, Role: LEADER I0517 06:44:04.155114 5527 leader_election.cc:379] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [CANDIDATE]: Term 1 election: Vote granted by peer 12bd807be9d84470a0d27627be106be4 (127.4.96.2:48598) I0517 06:44:04.154867 5788 consensus_queue.cc:204] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } I0517 06:44:04.159811 5099 catalog_manager.cc:3686] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf reported cstate change: term changed from 0 to 1, leader changed from to d6e1bdcef7c443299c40504db909f7cf (127.4.96.4). New cstate: current_term: 1 leader_uuid: "d6e1bdcef7c443299c40504db909f7cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } health_report { overall_health: UNKNOWN } } } I0517 06:44:04.174444 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:44:04.176636 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver db3928f6c5ea43bab4f27c1999b849db to finish bootstrapping I0517 06:44:04.186718 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 12bd807be9d84470a0d27627be106be4 to finish bootstrapping I0517 06:44:04.194947 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 896b9662512241d29d01130917359a60 to finish bootstrapping W0517 06:44:04.201902 5640 tablet.cc:1678] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:04.203122 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d6e1bdcef7c443299c40504db909f7cf to finish bootstrapping I0517 06:44:04.211283 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cb72eb5a50c24c5fa232a20acd802468 to finish bootstrapping I0517 06:44:04.221693 4480 kudu-admin-test.cc:376] Finding tablet leader and waiting for things to start... I0517 06:44:04.221822 4480 kudu-admin-test.cc:387] Waiting for Master to see the current replicas... I0517 06:44:04.224181 4480 kudu-admin-test.cc:395] Tablet locations: tablet_id: "9e4acfa8c48f4fbebd81cd58cfd3edc7" replicas { ts_info { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" rpc_addresses { host: "127.4.96.4" port: 56338 } } role: LEADER } replicas { ts_info { permanent_uuid: "12bd807be9d84470a0d27627be106be4" rpc_addresses { host: "127.4.96.2" port: 48598 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "896b9662512241d29d01130917359a60" rpc_addresses { host: "127.4.96.3" port: 44651 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } W0517 06:44:04.237804 5516 tablet.cc:1678] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:44:04.240962 5391 tablet.cc:1678] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:04.571641 5347 raft_consensus.cc:1057] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4 [term 1 FOLLOWER]: Refusing update from remote peer d6e1bdcef7c443299c40504db909f7cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:04.572779 5788 consensus_queue.cc:963] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:04.581513 5472 raft_consensus.cc:1057] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 1 FOLLOWER]: Refusing update from remote peer d6e1bdcef7c443299c40504db909f7cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:04.582800 5788 consensus_queue.cc:963] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:04.585871 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 5072 W0517 06:44:04.608599 5154 connection.cc:512] client connection to 127.4.96.62:49034 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:49034): Connection reset by peer (error 104) W0517 06:44:04.608613 5404 connection.cc:512] client connection to 127.4.96.62:49034 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) W0517 06:44:04.608620 5279 connection.cc:512] client connection to 127.4.96.62:49034 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:49034): Connection reset by peer (error 104) W0517 06:44:04.609344 5404 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0517 06:44:04.609755 5515 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:49034: Network error: Failed to send heartbeat to master: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) W0517 06:44:04.611522 5528 connection.cc:512] client connection to 127.4.96.62:49034 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:49034): Connection reset by peer (error 104) W0517 06:44:04.611527 5652 connection.cc:512] client connection to 127.4.96.62:49034 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:49034): Connection reset by peer (error 104) W0517 06:44:04.612519 5515 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:49034: Network error: Failed to ping master at 127.4.96.62:49034: Client connection negotiation failed: client connection to 127.4.96.62:49034: connect: Connection refused (error 111) W0517 06:44:04.612594 5071 connection.cc:512] client connection to 127.4.96.62:49034 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:49034): Connection reset by peer (error 104) W0517 06:44:04.613505 5528 connection.cc:512] server connection from 127.0.0.1:58382 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:58382): Connection reset by peer (error 104) W0517 06:44:04.613618 5770 connection.cc:512] client connection to 127.4.96.62:49034 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:49034): Connection reset by peer (error 104) W0517 06:44:04.614378 5278 connection.cc:512] server connection from 127.0.0.1:40600 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40600): Connection reset by peer (error 104) W0517 06:44:04.615118 5403 connection.cc:512] server connection from 127.0.0.1:39882 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39882): Connection reset by peer (error 104) I0517 06:44:04.615299 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 5518 W0517 06:44:04.633558 5071 connection.cc:512] client connection to 127.4.96.4:56338 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:56338): Connection reset by peer (error 104) W0517 06:44:04.633661 5281 connection.cc:512] server connection from 127.4.96.4:57740 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:57740): Connection reset by peer (error 104) W0517 06:44:04.634560 5406 connection.cc:512] server connection from 127.4.96.4:50299 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:50299): Connection reset by peer (error 104) W0517 06:44:04.634580 5070 connection.cc:512] client connection to 127.4.96.4:56338 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:56338): Connection reset by peer (error 104) I0517 06:44:04.634697 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 5269 W0517 06:44:04.661594 5071 connection.cc:512] client connection to 127.4.96.2:48598 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:48598): Connection reset by peer (error 104) W0517 06:44:04.661711 5068 connection.cc:512] client connection to 127.4.96.2:48598 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:48598): Connection reset by peer (error 104) I0517 06:44:04.662477 4480 kudu-admin-test.cc:413] Forcing unsafe config change on remaining follower 896b9662512241d29d01130917359a60 W0517 06:44:04.775394 5804 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:04.806109 5265 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:49034: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.96.62:49034: connect: Connection refused (error 111) I0517 06:44:04.930598 5763 heartbeater.cc:467] Master 127.4.96.62:49034 was elected leader, sending a full tablet report... W0517 06:44:04.934242 5763 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:49034: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.96.62:49034: connect: Connection refused (error 111) I0517 06:44:04.983126 5472 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "896b9662512241d29d01130917359a60" tablet_id: "9e4acfa8c48f4fbebd81cd58cfd3edc7" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "896b9662512241d29d01130917359a60" } } from {username='slave'} at 127.0.0.1:39892 W0517 06:44:04.983785 5472 raft_consensus.cc:1990] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 1 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } }NEW CONFIG: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } unsafe_config_change: true I0517 06:44:04.984344 5472 raft_consensus.cc:2756] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 1 FOLLOWER]: Advancing to term 2 W0517 06:44:04.988313 5515 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:49034: Network error: Failed to ping master at 127.4.96.62:49034: Client connection negotiation failed: client connection to 127.4.96.62:49034: connect: Connection refused (error 111) W0517 06:44:05.808354 5265 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:49034: Network error: Failed to ping master at 127.4.96.62:49034: Client connection negotiation failed: client connection to 127.4.96.62:49034: connect: Connection refused (error 111) W0517 06:44:05.936628 5763 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:49034: Network error: Failed to ping master at 127.4.96.62:49034: Client connection negotiation failed: client connection to 127.4.96.62:49034: connect: Connection refused (error 111) W0517 06:44:05.990528 5515 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:49034: Network error: Failed to ping master at 127.4.96.62:49034: Client connection negotiation failed: client connection to 127.4.96.62:49034: connect: Connection refused (error 111) I0517 06:44:06.489344 5824 raft_consensus.cc:435] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0517 06:44:06.489570 5824 raft_consensus.cc:2704] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Snoozing failure detection for 1.875s (starting election) I0517 06:44:06.489773 5824 raft_consensus.cc:457] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } unsafe_config_change: true I0517 06:44:06.490293 5824 leader_election.cc:270] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0517 06:44:06.491569 5825 raft_consensus.cc:2704] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Snoozing failure detection for 1.678s (election complete) I0517 06:44:06.491744 5825 raft_consensus.cc:2511] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0517 06:44:06.491869 5825 raft_consensus.cc:435] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0517 06:44:06.491997 5825 raft_consensus.cc:2704] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Snoozing failure detection for 2.096s (starting election) I0517 06:44:06.492118 5825 raft_consensus.cc:2756] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 2 FOLLOWER]: Advancing to term 3 I0517 06:44:06.494904 5825 raft_consensus.cc:457] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } unsafe_config_change: true I0517 06:44:06.495268 5825 leader_election.cc:270] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0517 06:44:06.495584 5824 raft_consensus.cc:2704] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Snoozing failure detection for 1.913s (election complete) I0517 06:44:06.495755 5824 raft_consensus.cc:2511] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 3 FOLLOWER]: Leader election won for term 3 I0517 06:44:06.495988 5824 raft_consensus.cc:571] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 3 LEADER]: Becoming Leader. State: Replica: 896b9662512241d29d01130917359a60, State: Running, Role: LEADER I0517 06:44:06.496314 5824 consensus_queue.cc:204] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } unsafe_config_change: true W0517 06:44:06.498615 5515 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:49034: Network error: Failed to ping master at 127.4.96.62:49034: Client connection negotiation failed: client connection to 127.4.96.62:49034: connect: Connection refused (error 111) I0517 06:44:06.499109 5824 raft_consensus.cc:2658] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 3 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 12bd807be9d84470a0d27627be106be4 (127.4.96.2) evicted, VOTER d6e1bdcef7c443299c40504db909f7cf (127.4.96.4) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } unsafe_config_change: true } W0517 06:44:06.503494 5515 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:49034: Network error: Failed to ping master at 127.4.96.62:49034: Client connection negotiation failed: client connection to 127.4.96.62:49034: connect: Connection refused (error 111) I0517 06:44:06.532387 4480 kudu-admin-test.cc:420] Restarting master... I0517 06:44:06.533035 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:49034 --webserver_interface=127.4.96.62 --webserver_port=56845 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:06.648352 5828 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:06.648681 5828 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:06.657578 5828 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:44:06.657709 5828 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:06.657827 5828 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:44:06.657965 5828 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:44:06.672112 5828 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:49034 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=56845 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:06.679780 5828 master_main.cc:80] Initializing master server... I0517 06:44:06.681257 5828 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 122319us I0517 06:44:06.682014 5828 fs_manager.cc:260] Metadata directory not provided I0517 06:44:06.682165 5828 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:06.687314 5828 fs_manager.cc:397] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s I0517 06:44:06.687943 5828 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:06.688338 5828 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:06.694741 5828 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.004s I0517 06:44:06.694859 5828 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "d199232303f34e778806ea6eeafe02cb" format_stamp: "Formatted at 2018-05-17 06:44:00 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:06.695976 5828 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0517 06:44:06.811386 5265 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:49034: Network error: Failed to ping master at 127.4.96.62:49034: Client connection negotiation failed: client connection to 127.4.96.62:49034: connect: Connection refused (error 111) I0517 06:44:06.872105 5828 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:06.873795 5828 master_main.cc:83] Starting Master server... I0517 06:44:06.929275 5828 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:49034 I0517 06:44:06.930166 5828 webserver.cc:173] Starting webserver on 127.4.96.62:56845 I0517 06:44:06.930272 5828 webserver.cc:184] Document root disabled I0517 06:44:06.931493 5828 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:56845/ I0517 06:44:06.934464 5828 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:44:06.936988 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 5828 I0517 06:44:06.945154 5890 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: Bootstrap starting. I0517 06:44:06.964720 5763 heartbeater.cc:346] Connected to a master server at 127.4.96.62:49034 I0517 06:44:06.964985 5763 heartbeater.cc:467] Master 127.4.96.62:49034 was elected leader, sending a full tablet report... W0517 06:44:06.966848 5763 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:49034: Service unavailable: Catalog manager is not initialized. State: 1 I0517 06:44:06.976152 5890 log.cc:520] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: Log is configured to *not* fsync() on all Append() calls I0517 06:44:06.987491 5890 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:06.988019 5890 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: Bootstrap complete. I0517 06:44:06.998448 5890 raft_consensus.cc:303] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } I0517 06:44:06.999992 5890 raft_consensus.cc:604] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d199232303f34e778806ea6eeafe02cb, State: Initialized, Role: FOLLOWER I0517 06:44:07.000450 5890 consensus_queue.cc:227] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } I0517 06:44:07.000779 5890 raft_consensus.cc:343] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:44:07.000916 5890 raft_consensus.cc:435] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:44:07.001065 5890 raft_consensus.cc:2704] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: Snoozing failure detection for 1.857s (starting election) I0517 06:44:07.001204 5890 raft_consensus.cc:2756] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 1 FOLLOWER]: Advancing to term 2 I0517 06:44:07.005679 5890 raft_consensus.cc:457] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } I0517 06:44:07.006125 5890 leader_election.cc:270] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0517 06:44:07.007452 5894 raft_consensus.cc:2704] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: Snoozing failure detection for 2.137s (election complete) I0517 06:44:07.007760 5894 raft_consensus.cc:2511] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 2 FOLLOWER]: Leader election won for term 2 I0517 06:44:07.009189 5894 raft_consensus.cc:571] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [term 2 LEADER]: Becoming Leader. State: Replica: d199232303f34e778806ea6eeafe02cb, State: Running, Role: LEADER I0517 06:44:07.009547 5894 consensus_queue.cc:204] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } I0517 06:44:07.012315 5890 sys_catalog.cc:335] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "d199232303f34e778806ea6eeafe02cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } } I0517 06:44:07.012701 5890 sys_catalog.cc:338] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: This master's current role is: LEADER I0517 06:44:07.014713 5890 sys_catalog.cc:422] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: configured and running, proceeding with master startup. I0517 06:44:07.014879 5898 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:44:07.015255 5896 sys_catalog.cc:335] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d199232303f34e778806ea6eeafe02cb. Latest consensus state: current_term: 2 leader_uuid: "d199232303f34e778806ea6eeafe02cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } } I0517 06:44:07.082836 5896 sys_catalog.cc:338] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: This master's current role is: LEADER I0517 06:44:07.015417 5895 sys_catalog.cc:335] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d199232303f34e778806ea6eeafe02cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d199232303f34e778806ea6eeafe02cb" member_type: VOTER } } I0517 06:44:07.085606 5828 master_main.cc:86] Master server successfully started. I0517 06:44:07.085680 5895 sys_catalog.cc:338] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb [sys.catalog]: This master's current role is: LEADER I0517 06:44:07.088994 5898 catalog_manager.cc:321] Loaded metadata for table TestTable [id=8f0321e0a36d4ee2bfb4d2c2a786bbf9] I0517 06:44:07.090919 5898 catalog_manager.cc:375] Loaded metadata for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (table TestTable [id=8f0321e0a36d4ee2bfb4d2c2a786bbf9]) I0517 06:44:07.091315 5898 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:44:07.094650 5898 catalog_manager.cc:1021] Loading token signing keys... I0517 06:44:07.096053 5898 catalog_manager.cc:3973] T 00000000000000000000000000000000 P d199232303f34e778806ea6eeafe02cb: Loaded TSK: 0 I0517 06:44:07.513824 5515 heartbeater.cc:346] Connected to a master server at 127.4.96.62:49034 I0517 06:44:07.515677 5857 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "896b9662512241d29d01130917359a60" instance_seqno: 1526539442962636) as {username='slave'} at 127.4.96.3:32994; Asking this server to re-register. I0517 06:44:07.516795 5515 heartbeater.cc:426] Registering TS with master... I0517 06:44:07.517082 5515 heartbeater.cc:475] Master 127.4.96.62:49034 requested a full tablet report, sending... I0517 06:44:07.518612 5857 ts_manager.cc:81] Registered new tserver with Master: 896b9662512241d29d01130917359a60 (127.4.96.3:44651) I0517 06:44:07.519549 5857 catalog_manager.cc:3686] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 reported cstate change: config changed from index -1 to 2, term changed from 1 to 3, leader changed from d6e1bdcef7c443299c40504db909f7cf (127.4.96.4) to 896b9662512241d29d01130917359a60 (127.4.96.3), VOTER 12bd807be9d84470a0d27627be106be4 (127.4.96.2) evicted, VOTER d6e1bdcef7c443299c40504db909f7cf (127.4.96.4) evicted. New cstate: current_term: 3 leader_uuid: "896b9662512241d29d01130917359a60" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:44:07.535830 5857 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf with a delay of 26 ms (attempt = 1) W0517 06:44:07.536067 5857 catalog_manager.cc:3845] Failed to send 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf: Not found: failed to reset TS proxy: Could not find TS for UUID d6e1bdcef7c443299c40504db909f7cf I0517 06:44:07.536294 5857 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 with a delay of 21 ms (attempt = 1) W0517 06:44:07.536497 5857 catalog_manager.cc:3845] Failed to send 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4: Not found: failed to reset TS proxy: Could not find TS for UUID 12bd807be9d84470a0d27627be106be4 I0517 06:44:07.537015 5857 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (attempt 1) W0517 06:44:07.537405 5857 catalog_manager.cc:3374] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 with cas_config_opid_index 2: no candidate replacement replica found for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (table TestTable [id=8f0321e0a36d4ee2bfb4d2c2a786bbf9]); the total number of registered tablet servers (1) does not allow for replacement of the failed replica: at least 4 tablet servers are required I0517 06:44:07.537617 5857 catalog_manager.cc:2791] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 with cas_config_opid_index 2 with a delay of 34 ms (attempt = 1) I0517 06:44:07.557945 5838 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 with a delay of 74 ms (attempt = 2) W0517 06:44:07.558130 5838 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 12bd807be9d84470a0d27627be106be4 I0517 06:44:07.562651 5836 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf with a delay of 64 ms (attempt = 2) W0517 06:44:07.562814 5836 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf failed: Not found: failed to reset TS proxy: Could not find TS for UUID d6e1bdcef7c443299c40504db909f7cf I0517 06:44:07.572289 5841 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (attempt 2) I0517 06:44:07.572607 5841 catalog_manager.cc:2791] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 with cas_config_opid_index 2 with a delay of 78 ms (attempt = 2) I0517 06:44:07.627477 5836 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf with a delay of 93 ms (attempt = 3) W0517 06:44:07.627768 5836 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf failed: Not found: failed to reset TS proxy: Could not find TS for UUID d6e1bdcef7c443299c40504db909f7cf I0517 06:44:07.632694 5838 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 with a delay of 85 ms (attempt = 3) W0517 06:44:07.632864 5838 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 12bd807be9d84470a0d27627be106be4 I0517 06:44:07.651172 5841 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (attempt 3) I0517 06:44:07.651486 5841 catalog_manager.cc:2791] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 with cas_config_opid_index 2 with a delay of 84 ms (attempt = 3) I0517 06:44:07.718479 5838 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 with a delay of 162 ms (attempt = 4) W0517 06:44:07.718650 5838 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 12bd807be9d84470a0d27627be106be4 I0517 06:44:07.721474 5836 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf with a delay of 150 ms (attempt = 4) W0517 06:44:07.721631 5836 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf failed: Not found: failed to reset TS proxy: Could not find TS for UUID d6e1bdcef7c443299c40504db909f7cf I0517 06:44:07.736160 5841 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (attempt 4) I0517 06:44:07.736454 5841 catalog_manager.cc:2791] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 with cas_config_opid_index 2 with a delay of 155 ms (attempt = 4) I0517 06:44:07.825660 5265 heartbeater.cc:346] Connected to a master server at 127.4.96.62:49034 I0517 06:44:07.826967 5857 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" instance_seqno: 1526539441726492) as {username='slave'} at 127.4.96.1:34395; Asking this server to re-register. I0517 06:44:07.827889 5265 heartbeater.cc:426] Registering TS with master... I0517 06:44:07.828128 5265 heartbeater.cc:475] Master 127.4.96.62:49034 requested a full tablet report, sending... I0517 06:44:07.829236 5857 ts_manager.cc:81] Registered new tserver with Master: db3928f6c5ea43bab4f27c1999b849db (127.4.96.1:53133) I0517 06:44:07.872329 5836 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf with a delay of 290 ms (attempt = 5) W0517 06:44:07.872587 5836 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf failed: Not found: failed to reset TS proxy: Could not find TS for UUID d6e1bdcef7c443299c40504db909f7cf I0517 06:44:07.881253 5838 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 with a delay of 296 ms (attempt = 5) W0517 06:44:07.881470 5838 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 12bd807be9d84470a0d27627be106be4 I0517 06:44:07.892364 5841 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (attempt 5) I0517 06:44:07.910959 5472 consensus_queue.cc:204] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:07.925246 5909 raft_consensus.cc:2658] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER db3928f6c5ea43bab4f27c1999b849db (127.4.96.1) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:07.933348 5841 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5) W0517 06:44:07.937191 5402 consensus_peers.cc:434] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 -> Peer db3928f6c5ea43bab4f27c1999b849db (127.4.96.1:53133): Couldn't send request to peer db3928f6c5ea43bab4f27c1999b849db for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e4acfa8c48f4fbebd81cd58cfd3edc7. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:07.937276 5857 catalog_manager.cc:3686] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 reported cstate change: config changed from index 2 to 4, NON_VOTER db3928f6c5ea43bab4f27c1999b849db (127.4.96.1) added. New cstate: current_term: 3 leader_uuid: "896b9662512241d29d01130917359a60" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0517 06:44:07.941109 5857 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (attempt 1) I0517 06:44:07.941372 5857 catalog_manager.cc:2791] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 with cas_config_opid_index 4 with a delay of 49 ms (attempt = 1) I0517 06:44:07.968485 5763 heartbeater.cc:346] Connected to a master server at 127.4.96.62:49034 I0517 06:44:07.968725 5763 heartbeater.cc:467] Master 127.4.96.62:49034 was elected leader, sending a full tablet report... I0517 06:44:07.969746 5857 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" instance_seqno: 1526539443882580) as {username='slave'} at 127.4.96.5:42783; Asking this server to re-register. I0517 06:44:07.970662 5763 heartbeater.cc:426] Registering TS with master... I0517 06:44:07.970886 5763 heartbeater.cc:475] Master 127.4.96.62:49034 requested a full tablet report, sending... I0517 06:44:07.971869 5857 ts_manager.cc:81] Registered new tserver with Master: cb72eb5a50c24c5fa232a20acd802468 (127.4.96.5:37703) I0517 06:44:07.991385 5836 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (attempt 2) I0517 06:44:07.992801 5472 consensus_peers.cc:456] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 -> Peer db3928f6c5ea43bab4f27c1999b849db (127.4.96.1:53133): Closing peer: db3928f6c5ea43bab4f27c1999b849db I0517 06:44:07.993193 5472 consensus_queue.cc:204] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: true } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: true } } unsafe_config_change: true W0517 06:44:07.996623 5402 consensus_peers.cc:434] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 -> Peer db3928f6c5ea43bab4f27c1999b849db (127.4.96.1:53133): Couldn't send request to peer db3928f6c5ea43bab4f27c1999b849db for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e4acfa8c48f4fbebd81cd58cfd3edc7. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:07.995489 5909 raft_consensus.cc:2658] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, NON_VOTER cb72eb5a50c24c5fa232a20acd802468 (127.4.96.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: true } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:08.002076 5841 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 2) I0517 06:44:08.004878 5857 catalog_manager.cc:3686] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 reported cstate change: config changed from index 4 to 5, NON_VOTER cb72eb5a50c24c5fa232a20acd802468 (127.4.96.5) added. New cstate: current_term: 3 leader_uuid: "896b9662512241d29d01130917359a60" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } W0517 06:44:08.012943 5404 consensus_peers.cc:434] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 -> Peer cb72eb5a50c24c5fa232a20acd802468 (127.4.96.5:37703): Couldn't send request to peer cb72eb5a50c24c5fa232a20acd802468 for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e4acfa8c48f4fbebd81cd58cfd3edc7. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:08.163281 5836 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf with a delay of 560 ms (attempt = 6) W0517 06:44:08.163465 5836 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf failed: Not found: failed to reset TS proxy: Could not find TS for UUID d6e1bdcef7c443299c40504db909f7cf I0517 06:44:08.178140 5838 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 with a delay of 542 ms (attempt = 6) W0517 06:44:08.178305 5838 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 12bd807be9d84470a0d27627be106be4 I0517 06:44:08.524379 5915 ts_tablet_manager.cc:669] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: Initiating tablet copy from peer 896b9662512241d29d01130917359a60 (127.4.96.3:44651) I0517 06:44:08.524674 5915 tablet_copy_client.cc:239] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.3:44651 I0517 06:44:08.529561 5918 ts_tablet_manager.cc:669] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: Initiating tablet copy from peer 896b9662512241d29d01130917359a60 (127.4.96.3:44651) I0517 06:44:08.529945 5918 tablet_copy_client.cc:239] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.3:44651 I0517 06:44:08.535097 5492 tablet_copy_service.cc:133] P 896b9662512241d29d01130917359a60: Received BeginTabletCopySession request for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 from peer db3928f6c5ea43bab4f27c1999b849db ({username='slave'} at 127.4.96.1:39827) I0517 06:44:08.535349 5492 tablet_copy_service.cc:154] P 896b9662512241d29d01130917359a60: Beginning new tablet copy session on tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 from peer db3928f6c5ea43bab4f27c1999b849db at {username='slave'} at 127.4.96.1:39827: session id = db3928f6c5ea43bab4f27c1999b849db-9e4acfa8c48f4fbebd81cd58cfd3edc7 I0517 06:44:08.538705 5492 tablet_copy_source_session.cc:202] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:44:08.541513 5915 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:08.542285 5492 tablet_copy_service.cc:133] P 896b9662512241d29d01130917359a60: Received BeginTabletCopySession request for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 from peer cb72eb5a50c24c5fa232a20acd802468 ({username='slave'} at 127.4.96.5:44297) I0517 06:44:08.542490 5492 tablet_copy_service.cc:154] P 896b9662512241d29d01130917359a60: Beginning new tablet copy session on tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 from peer cb72eb5a50c24c5fa232a20acd802468 at {username='slave'} at 127.4.96.5:44297: session id = cb72eb5a50c24c5fa232a20acd802468-9e4acfa8c48f4fbebd81cd58cfd3edc7 I0517 06:44:08.545159 5492 tablet_copy_source_session.cc:202] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:44:08.547745 5918 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:08.555632 5915 ts_tablet_manager.cc:1173] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:44:08.556073 5915 tablet_copy_client.cc:565] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: tablet copy: Starting download of 0 data blocks... I0517 06:44:08.556401 5915 tablet_copy_client.cc:528] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: tablet copy: Starting download of 1 WAL segments... I0517 06:44:08.558547 5915 tablet_copy_client.cc:404] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:44:08.562734 5915 ts_tablet_manager.cc:1028] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: Bootstrapping tablet I0517 06:44:08.562922 5918 ts_tablet_manager.cc:1173] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:44:08.563133 5915 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: Bootstrap starting. I0517 06:44:08.563354 5918 tablet_copy_client.cc:565] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: tablet copy: Starting download of 0 data blocks... I0517 06:44:08.563649 5918 tablet_copy_client.cc:528] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: tablet copy: Starting download of 1 WAL segments... I0517 06:44:08.565678 5918 tablet_copy_client.cc:404] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:44:08.569813 5918 ts_tablet_manager.cc:1028] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: Bootstrapping tablet I0517 06:44:08.570006 5915 log.cc:520] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: Log is configured to *not* fsync() on all Append() calls I0517 06:44:08.570263 5918 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: Bootstrap starting. I0517 06:44:08.576850 5918 log.cc:520] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: Log is configured to *not* fsync() on all Append() calls I0517 06:44:08.577291 5915 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:08.577883 5915 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: Bootstrap complete. I0517 06:44:08.578258 5915 ts_tablet_manager.cc:1045] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.008s I0517 06:44:08.584112 5918 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:08.584679 5918 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: Bootstrap complete. I0517 06:44:08.585081 5918 ts_tablet_manager.cc:1045] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s I0517 06:44:08.592866 5915 raft_consensus.cc:303] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: true } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:08.593648 5915 raft_consensus.cc:604] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: db3928f6c5ea43bab4f27c1999b849db, State: Initialized, Role: LEARNER I0517 06:44:08.594081 5915 consensus_queue.cc:227] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: true } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:08.599453 5918 raft_consensus.cc:303] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: true } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:08.600178 5918 raft_consensus.cc:604] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: cb72eb5a50c24c5fa232a20acd802468, State: Initialized, Role: LEARNER I0517 06:44:08.600615 5918 consensus_queue.cc:227] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: true } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:08.606163 5915 ts_tablet_manager.cc:1073] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: Time spent starting tablet: real 0.028s user 0.016s sys 0.012s I0517 06:44:08.607880 5918 ts_tablet_manager.cc:1073] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: Time spent starting tablet: real 0.023s user 0.024s sys 0.004s I0517 06:44:08.609073 5491 tablet_copy_service.cc:323] P 896b9662512241d29d01130917359a60: Request end of tablet copy session cb72eb5a50c24c5fa232a20acd802468-9e4acfa8c48f4fbebd81cd58cfd3edc7 received from {username='slave'} at 127.4.96.5:44297 I0517 06:44:08.609302 5491 tablet_copy_service.cc:411] P 896b9662512241d29d01130917359a60: Ending tablet copy session cb72eb5a50c24c5fa232a20acd802468-9e4acfa8c48f4fbebd81cd58cfd3edc7 on tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 with peer cb72eb5a50c24c5fa232a20acd802468 I0517 06:44:08.609844 5492 tablet_copy_service.cc:323] P 896b9662512241d29d01130917359a60: Request end of tablet copy session db3928f6c5ea43bab4f27c1999b849db-9e4acfa8c48f4fbebd81cd58cfd3edc7 received from {username='slave'} at 127.4.96.1:39827 I0517 06:44:08.610128 5492 tablet_copy_service.cc:411] P 896b9662512241d29d01130917359a60: Ending tablet copy session db3928f6c5ea43bab4f27c1999b849db-9e4acfa8c48f4fbebd81cd58cfd3edc7 on tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 with peer db3928f6c5ea43bab4f27c1999b849db W0517 06:44:08.660502 5764 tablet.cc:1678] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:08.721195 5838 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 with a delay of 1037 ms (attempt = 7) W0517 06:44:08.721371 5838 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 12bd807be9d84470a0d27627be106be4 I0517 06:44:08.724354 5836 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf with a delay of 1071 ms (attempt = 7) W0517 06:44:08.724535 5836 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf failed: Not found: failed to reset TS proxy: Could not find TS for UUID d6e1bdcef7c443299c40504db909f7cf W0517 06:44:08.773902 5266 tablet.cc:1678] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:09.036728 5720 raft_consensus.cc:1002] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.4->[3.5-3.5] Dedup: 3.5->[] I0517 06:44:09.112095 5222 raft_consensus.cc:1002] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db [term 3 LEARNER]: Deduplicated request from leader. Original: 3.4->[3.5-3.5] Dedup: 3.5->[] I0517 06:44:09.526902 5927 raft_consensus.cc:866] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: attempting to promote NON_VOTER db3928f6c5ea43bab4f27c1999b849db to VOTER I0517 06:44:09.527508 5927 consensus_peers.cc:456] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 -> Peer cb72eb5a50c24c5fa232a20acd802468 (127.4.96.5:37703): Closing peer: cb72eb5a50c24c5fa232a20acd802468 I0517 06:44:09.527643 5927 consensus_peers.cc:456] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 -> Peer db3928f6c5ea43bab4f27c1999b849db (127.4.96.1:53133): Closing peer: db3928f6c5ea43bab4f27c1999b849db I0517 06:44:09.528038 5927 consensus_queue.cc:204] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: false } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:09.531165 5222 raft_consensus.cc:1057] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db [term 3 LEARNER]: Refusing update from remote peer 896b9662512241d29d01130917359a60: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch) I0517 06:44:09.531347 5720 raft_consensus.cc:1057] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468 [term 3 LEARNER]: Refusing update from remote peer 896b9662512241d29d01130917359a60: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch) I0517 06:44:09.532239 5926 consensus_queue.cc:963] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0517 06:44:09.532693 5925 consensus_queue.cc:963] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0517 06:44:09.546783 5927 raft_consensus.cc:829] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 3 LEADER]: attempt to promote peer cb72eb5a50c24c5fa232a20acd802468: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing. I0517 06:44:09.547052 5925 raft_consensus.cc:2658] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, db3928f6c5ea43bab4f27c1999b849db (127.4.96.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: false } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:09.548307 5222 raft_consensus.cc:2658] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, db3928f6c5ea43bab4f27c1999b849db (127.4.96.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: false } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:09.552256 5720 raft_consensus.cc:2658] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468 [term 3 LEARNER]: Committing config change with OpId 3.6: config changed from index 5 to 6, db3928f6c5ea43bab4f27c1999b849db (127.4.96.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: false } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:09.554538 5857 catalog_manager.cc:3686] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db reported cstate change: config changed from index 5 to 6, db3928f6c5ea43bab4f27c1999b849db (127.4.96.1) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "896b9662512241d29d01130917359a60" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: false } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:09.560241 5925 raft_consensus.cc:866] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60: attempting to promote NON_VOTER cb72eb5a50c24c5fa232a20acd802468 to VOTER I0517 06:44:09.561028 5925 consensus_peers.cc:456] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 -> Peer cb72eb5a50c24c5fa232a20acd802468 (127.4.96.5:37703): Closing peer: cb72eb5a50c24c5fa232a20acd802468 I0517 06:44:09.561249 5925 consensus_peers.cc:456] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 -> Peer db3928f6c5ea43bab4f27c1999b849db (127.4.96.1:53133): Closing peer: db3928f6c5ea43bab4f27c1999b849db I0517 06:44:09.561766 5925 consensus_queue.cc:204] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: false } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: false } } unsafe_config_change: true I0517 06:44:09.564786 5720 raft_consensus.cc:1057] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468 [term 3 LEARNER]: Refusing update from remote peer 896b9662512241d29d01130917359a60: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:44:09.565872 5222 raft_consensus.cc:1057] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db [term 3 FOLLOWER]: Refusing update from remote peer 896b9662512241d29d01130917359a60: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:44:09.566890 5926 consensus_queue.cc:963] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:44:09.567337 5925 consensus_queue.cc:963] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:44:09.570807 5926 raft_consensus.cc:2658] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 896b9662512241d29d01130917359a60 [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, cb72eb5a50c24c5fa232a20acd802468 (127.4.96.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: false } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:09.572093 5222 raft_consensus.cc:2658] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, cb72eb5a50c24c5fa232a20acd802468 (127.4.96.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: false } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:09.572060 5720 raft_consensus.cc:2658] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P cb72eb5a50c24c5fa232a20acd802468 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, cb72eb5a50c24c5fa232a20acd802468 (127.4.96.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: false } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:09.577038 5855 catalog_manager.cc:3686] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P db3928f6c5ea43bab4f27c1999b849db reported cstate change: config changed from index 6 to 7, cb72eb5a50c24c5fa232a20acd802468 (127.4.96.5) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "896b9662512241d29d01130917359a60" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } peers { permanent_uuid: "db3928f6c5ea43bab4f27c1999b849db" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 53133 } attrs { promote: false } } peers { permanent_uuid: "cb72eb5a50c24c5fa232a20acd802468" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 37703 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:09.715658 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:56338 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_port=46665 --webserver_interface=127.4.96.4 --tserver_master_addrs=127.4.96.62:49034 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:09.759016 5838 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 with a delay of 2048 ms (attempt = 8) W0517 06:44:09.759212 5838 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=12bd807be9d84470a0d27627be106be4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 12bd807be9d84470a0d27627be106be4 I0517 06:44:09.796701 5836 catalog_manager.cc:2791] Scheduling retry of 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf with a delay of 2078 ms (attempt = 8) W0517 06:44:09.796898 5836 catalog_manager.cc:2810] Async tablet task 9e4acfa8c48f4fbebd81cd58cfd3edc7 Delete Tablet RPC for TS=d6e1bdcef7c443299c40504db909f7cf failed: Not found: failed to reset TS proxy: Could not find TS for UUID d6e1bdcef7c443299c40504db909f7cf WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:09.818393 5942 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:09.818693 5942 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:09.826681 5942 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:09.826995 5942 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:44:09.839599 5942 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:56338 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=46665 --tserver_master_addrs=127.4.96.62:49034 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:09.847615 5942 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:09.849354 5942 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 123819us I0517 06:44:09.849774 5942 fs_manager.cc:260] Metadata directory not provided I0517 06:44:09.849900 5942 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:09.855155 5942 fs_manager.cc:397] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s I0517 06:44:09.855726 5942 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:09.856133 5942 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:09.862586 5942 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:44:09.862695 5942 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "d6e1bdcef7c443299c40504db909f7cf" format_stamp: "Formatted at 2018-05-17 06:44:03 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:09.863826 5942 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:10.043680 5942 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:10.046039 5942 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0517 06:44:10.046159 5942 ts_tablet_manager.cc:1005] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Loading tablet metadata I0517 06:44:10.050179 5942 ts_tablet_manager.cc:356] Loaded tablet metadata (1 live tablets) I0517 06:44:10.053007 5942 ts_tablet_manager.cc:1173] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:10.056398 5959 ts_tablet_manager.cc:1028] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Bootstrapping tablet I0517 06:44:10.056901 5959 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Bootstrap starting. I0517 06:44:10.056737 5942 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:10.081660 5959 log.cc:520] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Log is configured to *not* fsync() on all Append() calls I0517 06:44:10.086742 5959 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:10.087469 5959 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Bootstrap complete. I0517 06:44:10.088730 5959 ts_tablet_manager.cc:1045] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.012s I0517 06:44:10.111793 5959 raft_consensus.cc:303] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } I0517 06:44:10.118046 5959 raft_consensus.cc:604] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e1bdcef7c443299c40504db909f7cf, State: Initialized, Role: FOLLOWER I0517 06:44:10.118690 5959 consensus_queue.cc:227] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } I0517 06:44:10.127106 5959 ts_tablet_manager.cc:1073] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Time spent starting tablet: real 0.038s user 0.032s sys 0.000s I0517 06:44:10.187821 5942 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:56338 I0517 06:44:10.188674 5942 webserver.cc:173] Starting webserver on 127.4.96.4:46665 I0517 06:44:10.188771 5942 webserver.cc:184] Document root disabled I0517 06:44:10.189966 5942 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:46665/ I0517 06:44:10.190614 5942 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:44:10.193614 5942 tablet_server_main.cc:93] Tablet server successfully started. W0517 06:44:10.194577 6067 tablet.cc:1678] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:10.200760 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 5942 I0517 06:44:10.201824 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:48598 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_port=55046 --webserver_interface=127.4.96.2 --tserver_master_addrs=127.4.96.62:49034 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:10.203593 6066 heartbeater.cc:346] Connected to a master server at 127.4.96.62:49034 I0517 06:44:10.203829 6066 heartbeater.cc:426] Registering TS with master... I0517 06:44:10.204265 6066 heartbeater.cc:475] Master 127.4.96.62:49034 requested a full tablet report, sending... I0517 06:44:10.205883 5857 ts_manager.cc:81] Registered new tserver with Master: d6e1bdcef7c443299c40504db909f7cf (127.4.96.4:56338) I0517 06:44:10.206734 5857 catalog_manager.cc:3013] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 on d6e1bdcef7c443299c40504db909f7cf (127.4.96.4:56338) (Replica has no consensus available (current committed config index is 7)) I0517 06:44:10.208770 5857 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:55305 I0517 06:44:10.219663 6003 tablet_service.cc:800] Processing DeleteTablet for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 7)) from {username='slave'} at 127.0.0.1:58399 I0517 06:44:10.221577 6073 tablet_replica.cc:261] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: stopping tablet replica I0517 06:44:10.221788 6073 maintenance_manager.cc:237] P d6e1bdcef7c443299c40504db909f7cf: Unregistered op CompactRowSetsOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.221953 6073 maintenance_manager.cc:237] P d6e1bdcef7c443299c40504db909f7cf: Unregistered op MinorDeltaCompactionOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.222081 6073 maintenance_manager.cc:237] P d6e1bdcef7c443299c40504db909f7cf: Unregistered op MajorDeltaCompactionOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.222205 6073 maintenance_manager.cc:237] P d6e1bdcef7c443299c40504db909f7cf: Unregistered op UndoDeltaBlockGCOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.222383 6073 maintenance_manager.cc:237] P d6e1bdcef7c443299c40504db909f7cf: Unregistered op FlushMRSOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.222502 6073 maintenance_manager.cc:237] P d6e1bdcef7c443299c40504db909f7cf: Unregistered op FlushDeltaMemStoresOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.222631 6073 maintenance_manager.cc:237] P d6e1bdcef7c443299c40504db909f7cf: Unregistered op LogGCOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.222800 6073 raft_consensus.cc:2016] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 1 FOLLOWER]: Raft consensus shutting down. I0517 06:44:10.223048 6073 raft_consensus.cc:2043] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf [term 1 FOLLOWER]: Raft consensus is shut down! I0517 06:44:10.224665 6073 ts_tablet_manager.cc:1369] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0517 06:44:10.230918 6073 ts_tablet_manager.cc:1382] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0517 06:44:10.231088 6073 log.cc:981] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P d6e1bdcef7c443299c40504db909f7cf: Deleting WAL directory at /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/wals/9e4acfa8c48f4fbebd81cd58cfd3edc7 I0517 06:44:10.232278 5841 catalog_manager.cc:2991] TS d6e1bdcef7c443299c40504db909f7cf (127.4.96.4:56338): tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (table TestTable [id=8f0321e0a36d4ee2bfb4d2c2a786bbf9]) successfully deleted WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:10.310338 6070 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:10.310639 6070 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:10.318667 6070 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:10.318984 6070 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:44:10.331320 6070 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:48598 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=55046 --tserver_master_addrs=127.4.96.62:49034 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:10.339294 6070 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:10.341040 6070 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 124319us I0517 06:44:10.341456 6070 fs_manager.cc:260] Metadata directory not provided I0517 06:44:10.341580 6070 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:10.346743 6070 fs_manager.cc:397] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s I0517 06:44:10.347326 6070 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:10.347724 6070 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:10.354303 6070 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0517 06:44:10.354411 6070 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "12bd807be9d84470a0d27627be106be4" format_stamp: "Formatted at 2018-05-17 06:44:01 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:10.355507 6070 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:10.500898 6070 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:10.503293 6070 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0517 06:44:10.503414 6070 ts_tablet_manager.cc:1005] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Loading tablet metadata I0517 06:44:10.507555 6070 ts_tablet_manager.cc:356] Loaded tablet metadata (1 live tablets) I0517 06:44:10.510483 6070 ts_tablet_manager.cc:1173] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:10.513756 6090 ts_tablet_manager.cc:1028] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Bootstrapping tablet I0517 06:44:10.514273 6090 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Bootstrap starting. I0517 06:44:10.514119 6070 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:10.538616 6090 log.cc:520] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Log is configured to *not* fsync() on all Append() calls I0517 06:44:10.544235 6090 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:10.545025 6090 tablet_bootstrap.cc:436] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Bootstrap complete. I0517 06:44:10.546288 6090 ts_tablet_manager.cc:1045] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.000s I0517 06:44:10.579844 6090 raft_consensus.cc:303] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } I0517 06:44:10.588963 6090 raft_consensus.cc:604] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bd807be9d84470a0d27627be106be4, State: Initialized, Role: FOLLOWER I0517 06:44:10.589952 6090 consensus_queue.cc:227] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6e1bdcef7c443299c40504db909f7cf" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56338 } } peers { permanent_uuid: "12bd807be9d84470a0d27627be106be4" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48598 } } peers { permanent_uuid: "896b9662512241d29d01130917359a60" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 44651 } } I0517 06:44:10.594350 6090 ts_tablet_manager.cc:1073] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Time spent starting tablet: real 0.048s user 0.036s sys 0.008s I0517 06:44:10.659953 6070 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:48598 I0517 06:44:10.660833 6070 webserver.cc:173] Starting webserver on 127.4.96.2:55046 I0517 06:44:10.660938 6070 webserver.cc:184] Document root disabled I0517 06:44:10.662688 6070 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:55046/ I0517 06:44:10.663328 6070 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:44:10.668263 6070 tablet_server_main.cc:93] Tablet server successfully started. W0517 06:44:10.668493 6198 tablet.cc:1678] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:10.669888 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 6070 I0517 06:44:10.679473 6197 heartbeater.cc:346] Connected to a master server at 127.4.96.62:49034 I0517 06:44:10.679697 6197 heartbeater.cc:426] Registering TS with master... I0517 06:44:10.680099 6197 heartbeater.cc:475] Master 127.4.96.62:49034 requested a full tablet report, sending... I0517 06:44:10.681651 5857 ts_manager.cc:81] Registered new tserver with Master: 12bd807be9d84470a0d27627be106be4 (127.4.96.2:48598) I0517 06:44:10.682461 5857 catalog_manager.cc:3013] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 on 12bd807be9d84470a0d27627be106be4 (127.4.96.2:48598) (Replica has no consensus available (current committed config index is 7)) I0517 06:44:10.684990 5857 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:54495 I0517 06:44:10.699908 6133 tablet_service.cc:800] Processing DeleteTablet for tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 7)) from {username='slave'} at 127.0.0.1:40618 I0517 06:44:10.702616 6203 tablet_replica.cc:261] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: stopping tablet replica I0517 06:44:10.702847 6203 maintenance_manager.cc:237] P 12bd807be9d84470a0d27627be106be4: Unregistered op CompactRowSetsOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.702987 6203 maintenance_manager.cc:237] P 12bd807be9d84470a0d27627be106be4: Unregistered op MinorDeltaCompactionOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.703104 6203 maintenance_manager.cc:237] P 12bd807be9d84470a0d27627be106be4: Unregistered op MajorDeltaCompactionOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.703217 6203 maintenance_manager.cc:237] P 12bd807be9d84470a0d27627be106be4: Unregistered op UndoDeltaBlockGCOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.703377 6203 maintenance_manager.cc:237] P 12bd807be9d84470a0d27627be106be4: Unregistered op FlushMRSOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.703500 6203 maintenance_manager.cc:237] P 12bd807be9d84470a0d27627be106be4: Unregistered op FlushDeltaMemStoresOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.703629 6203 maintenance_manager.cc:237] P 12bd807be9d84470a0d27627be106be4: Unregistered op LogGCOp(9e4acfa8c48f4fbebd81cd58cfd3edc7) I0517 06:44:10.703799 6203 raft_consensus.cc:2016] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4 [term 1 FOLLOWER]: Raft consensus shutting down. I0517 06:44:10.704043 6203 raft_consensus.cc:2043] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4 [term 1 FOLLOWER]: Raft consensus is shut down! I0517 06:44:10.705586 6203 ts_tablet_manager.cc:1369] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0517 06:44:10.712357 6203 ts_tablet_manager.cc:1382] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0517 06:44:10.712530 6203 log.cc:981] T 9e4acfa8c48f4fbebd81cd58cfd3edc7 P 12bd807be9d84470a0d27627be106be4: Deleting WAL directory at /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/wals/9e4acfa8c48f4fbebd81cd58cfd3edc7 I0517 06:44:10.713549 5837 catalog_manager.cc:2991] TS 12bd807be9d84470a0d27627be106be4 (127.4.96.2:48598): tablet 9e4acfa8c48f4fbebd81cd58cfd3edc7 (table TestTable [id=8f0321e0a36d4ee2bfb4d2c2a786bbf9]) successfully deleted I0517 06:44:10.715865 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 5134 W0517 06:44:10.735904 5071 connection.cc:512] client connection to 127.4.96.1:53133 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:53133): Connection reset by peer (error 104) W0517 06:44:10.735939 5841 connection.cc:512] server connection from 127.4.96.1:34395 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:34395): Connection reset by peer (error 104) W0517 06:44:10.736891 5402 connection.cc:512] client connection to 127.4.96.1:53133 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:53133): Connection reset by peer (error 104) W0517 06:44:10.737829 5404 connection.cc:512] server connection from 127.4.96.1:39827 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:39827): Connection reset by peer (error 104) W0517 06:44:10.737820 5067 connection.cc:512] client connection to 127.4.96.1:53133 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:53133): Connection reset by peer (error 104) I0517 06:44:10.738708 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 6070 W0517 06:44:10.754825 5837 connection.cc:512] client connection to 127.4.96.2:48598 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:48598): Connection reset by peer (error 104) W0517 06:44:10.754874 5836 connection.cc:512] server connection from 127.4.96.2:54495 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:54495): Connection reset by peer (error 104) W0517 06:44:10.755502 5068 connection.cc:512] client connection to 127.4.96.2:48598 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:48598): Connection reset by peer (error 104) I0517 06:44:10.755640 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 5394 W0517 06:44:10.781935 5836 connection.cc:512] server connection from 127.4.96.3:32994 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:32994): Connection reset by peer (error 104) W0517 06:44:10.781985 5071 connection.cc:512] client connection to 127.4.96.3:44651 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:44651): Connection reset by peer (error 104) W0517 06:44:10.782022 5652 connection.cc:512] server connection from 127.4.96.3:47317 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:47317): Connection reset by peer (error 104) W0517 06:44:10.782691 5070 connection.cc:512] client connection to 127.4.96.3:44651 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:44651): Connection reset by peer (error 104) W0517 06:44:10.782876 5654 connection.cc:512] client connection to 127.4.96.3:44651 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:44651): Connection reset by peer (error 104) I0517 06:44:10.783619 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 5942 W0517 06:44:10.794140 5841 connection.cc:512] client connection to 127.4.96.3:44651 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:44651): Connection reset by peer (error 104) W0517 06:44:10.803352 5841 connection.cc:512] client connection to 127.4.96.4:56338 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:56338): Connection reset by peer (error 104) W0517 06:44:10.803408 5070 connection.cc:512] client connection to 127.4.96.4:56338 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:56338): Connection reset by peer (error 104) W0517 06:44:10.804108 5838 connection.cc:512] server connection from 127.4.96.4:55305 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:55305): Connection reset by peer (error 104) I0517 06:44:10.804728 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 5642 W0517 06:44:10.828783 5071 connection.cc:512] client connection to 127.4.96.5:37703 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:37703): Connection reset by peer (error 104) W0517 06:44:10.828802 5838 connection.cc:512] server connection from 127.4.96.5:42783 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:42783): Connection reset by peer (error 104) W0517 06:44:10.828903 5069 connection.cc:512] client connection to 127.4.96.5:37703 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:37703): Connection reset by peer (error 104) I0517 06:44:10.829949 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 5828 W0517 06:44:10.855593 5071 connection.cc:512] client connection to 127.4.96.62:49034 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:49034): Connection reset by peer (error 104) [ OK ] AdminCliTest.TestUnsafeChangeConfigOnSingleFollower (10055 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigOnSingleLeader I0517 06:44:10.870883 4480 test_util.cc:200] Using random seed: 699690062 I0517 06:44:10.873860 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:44:10.873975 4480 ts_itest-base.cc:114] -------------- I0517 06:44:10.874053 4480 ts_itest-base.cc:115] 5 tablet servers I0517 06:44:10.874127 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:44:10.874200 4480 ts_itest-base.cc:117] -------------- I0517 06:44:10.876250 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:39669 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:10.987772 6209 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:10.988088 6209 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:10.996978 6209 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:44:10.997097 6209 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:10.997193 6209 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:44:10.997303 6209 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:44:11.011108 6209 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:39669 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:11.018779 6209 master_main.cc:80] Initializing master server... I0517 06:44:11.020264 6209 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 124819us I0517 06:44:11.021003 6209 fs_manager.cc:260] Metadata directory not provided I0517 06:44:11.021124 6209 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:11.021317 6209 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:44:11.021425 6209 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:11.028622 6209 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "354b55e91977404abb71f4ca603fe80a" format_stamp: "Formatted at 2018-05-17 06:44:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:11.029270 6209 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "354b55e91977404abb71f4ca603fe80a" format_stamp: "Formatted at 2018-05-17 06:44:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:11.038761 6209 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:44:11.039918 6209 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:11.040318 6209 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:11.046710 6209 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0517 06:44:11.046818 6209 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "354b55e91977404abb71f4ca603fe80a" format_stamp: "Formatted at 2018-05-17 06:44:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:11.047011 6209 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:11.225960 6209 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:11.227630 6209 master_main.cc:83] Starting Master server... I0517 06:44:11.266844 6209 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:39669 I0517 06:44:11.267663 6209 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:44:11.267760 6209 webserver.cc:184] Document root disabled I0517 06:44:11.269085 6209 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:51907/ I0517 06:44:11.269803 6209 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:44:11.272423 6270 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:11.278412 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 6209 I0517 06:44:11.279723 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:39669 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:11.288164 6270 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: Bootstrap starting. I0517 06:44:11.290396 6270 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: No blocks or log segments found. Creating new log. I0517 06:44:11.291532 6270 log.cc:520] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: Log is configured to *not* fsync() on all Append() calls I0517 06:44:11.293800 6270 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: No bootstrap required, opened a new log I0517 06:44:11.305397 6270 raft_consensus.cc:303] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } I0517 06:44:11.305693 6270 raft_consensus.cc:329] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:11.305820 6270 raft_consensus.cc:604] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354b55e91977404abb71f4ca603fe80a, State: Initialized, Role: FOLLOWER I0517 06:44:11.306265 6270 consensus_queue.cc:227] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } I0517 06:44:11.306605 6270 raft_consensus.cc:343] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:44:11.306741 6270 raft_consensus.cc:435] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:44:11.306886 6270 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: Snoozing failure detection for 1.645s (starting election) I0517 06:44:11.307023 6270 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:11.309412 6270 raft_consensus.cc:457] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } I0517 06:44:11.309813 6270 leader_election.cc:270] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:44:11.311389 6273 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: Snoozing failure detection for 1.620s (election complete) I0517 06:44:11.311661 6273 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:44:11.322419 6273 raft_consensus.cc:571] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 1 LEADER]: Becoming Leader. State: Replica: 354b55e91977404abb71f4ca603fe80a, State: Running, Role: LEADER I0517 06:44:11.322823 6273 consensus_queue.cc:204] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } I0517 06:44:11.329747 6270 sys_catalog.cc:335] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "354b55e91977404abb71f4ca603fe80a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } } I0517 06:44:11.330168 6270 sys_catalog.cc:338] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: This master's current role is: LEADER I0517 06:44:11.332499 6270 sys_catalog.cc:422] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: configured and running, proceeding with master startup. I0517 06:44:11.331636 6275 sys_catalog.cc:335] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 354b55e91977404abb71f4ca603fe80a. Latest consensus state: current_term: 1 leader_uuid: "354b55e91977404abb71f4ca603fe80a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } } I0517 06:44:11.391575 6275 sys_catalog.cc:338] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: This master's current role is: LEADER I0517 06:44:11.331899 6274 sys_catalog.cc:335] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "354b55e91977404abb71f4ca603fe80a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } } I0517 06:44:11.393525 6274 sys_catalog.cc:338] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: This master's current role is: LEADER I0517 06:44:11.398927 6209 master_main.cc:86] Master server successfully started. W0517 06:44:11.401779 6278 catalog_manager.cc:1047] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0517 06:44:11.402025 6278 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0517 06:44:11.402287 6277 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:44:11.404392 6277 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:11.420225 6271 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:11.420570 6271 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:11.428467 6271 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:11.428791 6271 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:44:11.440269 6277 catalog_manager.cc:899] Generated new certificate authority record I0517 06:44:11.442473 6277 catalog_manager.cc:1021] Loading token signing keys... I0517 06:44:11.445341 6271 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:39669 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:11.458225 6271 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:11.460539 6271 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 124819us I0517 06:44:11.461107 6271 fs_manager.cc:260] Metadata directory not provided I0517 06:44:11.461274 6271 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:11.461563 6271 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:44:11.461735 6271 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:11.473834 6271 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "f9766287bc0d4d29927a018255e773e1" format_stamp: "Formatted at 2018-05-17 06:44:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:11.474287 6277 catalog_manager.cc:3962] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: Generated new TSK 0 I0517 06:44:11.474820 6271 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "f9766287bc0d4d29927a018255e773e1" format_stamp: "Formatted at 2018-05-17 06:44:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:11.485075 6271 fs_manager.cc:495] Time spent creating directory manager: real 0.010s user 0.004s sys 0.004s I0517 06:44:11.486186 6271 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:11.486604 6271 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:11.493408 6271 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:44:11.493537 6271 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "f9766287bc0d4d29927a018255e773e1" format_stamp: "Formatted at 2018-05-17 06:44:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:11.493753 6271 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:11.710553 6271 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:11.713032 6271 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:11.714203 6271 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:11.816089 6271 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:33942 I0517 06:44:11.816977 6271 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:44:11.817083 6271 webserver.cc:184] Document root disabled I0517 06:44:11.818467 6271 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:53940/ I0517 06:44:11.819177 6271 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:44:11.821768 6271 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:11.826153 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 6271 I0517 06:44:11.827392 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:39669 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:11.848222 6402 heartbeater.cc:346] Connected to a master server at 127.4.96.62:39669 I0517 06:44:11.848507 6402 heartbeater.cc:426] Registering TS with master... I0517 06:44:11.849047 6402 heartbeater.cc:475] Master 127.4.96.62:39669 requested a full tablet report, sending... I0517 06:44:11.851132 6237 ts_manager.cc:81] Registered new tserver with Master: f9766287bc0d4d29927a018255e773e1 (127.4.96.1:33942) I0517 06:44:11.853668 6237 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:50794 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:11.935811 6406 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:11.936166 6406 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:11.944084 6406 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:11.944408 6406 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:44:11.957188 6406 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:39669 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:11.965240 6406 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:11.967015 6406 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 124819us I0517 06:44:11.967449 6406 fs_manager.cc:260] Metadata directory not provided I0517 06:44:11.967577 6406 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:11.967775 6406 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:44:11.967895 6406 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:11.976981 6406 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "b381de9177314b1294f8324c1ea8ef96" format_stamp: "Formatted at 2018-05-17 06:44:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:11.977771 6406 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "b381de9177314b1294f8324c1ea8ef96" format_stamp: "Formatted at 2018-05-17 06:44:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:11.987577 6406 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:44:11.988656 6406 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:11.989040 6406 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:11.995683 6406 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:44:11.995796 6406 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "b381de9177314b1294f8324c1ea8ef96" format_stamp: "Formatted at 2018-05-17 06:44:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:11.996024 6406 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:12.200248 6406 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:12.202857 6406 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:12.204018 6406 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:12.334276 6406 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:35758 I0517 06:44:12.335232 6406 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:44:12.335337 6406 webserver.cc:184] Document root disabled I0517 06:44:12.336750 6406 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:39407/ I0517 06:44:12.337422 6406 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:44:12.340149 6406 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:12.345108 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 6406 I0517 06:44:12.346047 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:39669 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:12.366067 6527 heartbeater.cc:346] Connected to a master server at 127.4.96.62:39669 I0517 06:44:12.366351 6527 heartbeater.cc:426] Registering TS with master... I0517 06:44:12.366890 6527 heartbeater.cc:475] Master 127.4.96.62:39669 requested a full tablet report, sending... I0517 06:44:12.368525 6237 ts_manager.cc:81] Registered new tserver with Master: b381de9177314b1294f8324c1ea8ef96 (127.4.96.2:35758) I0517 06:44:12.370240 6237 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:54254 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:12.451012 6530 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:12.451350 6530 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:12.459296 6530 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:12.459619 6530 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:44:12.472623 6530 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:39669 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:12.480772 6530 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:12.482545 6530 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 125319us I0517 06:44:12.482966 6530 fs_manager.cc:260] Metadata directory not provided I0517 06:44:12.483093 6530 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:12.483289 6530 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:44:12.483407 6530 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:12.490989 6530 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "c95723aafe594cc999bfe59b9aee2b1e" format_stamp: "Formatted at 2018-05-17 06:44:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:12.491667 6530 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "c95723aafe594cc999bfe59b9aee2b1e" format_stamp: "Formatted at 2018-05-17 06:44:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:12.501372 6530 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:44:12.502502 6530 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:12.502898 6530 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:12.509601 6530 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:44:12.509711 6530 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "c95723aafe594cc999bfe59b9aee2b1e" format_stamp: "Formatted at 2018-05-17 06:44:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:12.509954 6530 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:12.744105 6530 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:12.746418 6530 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:12.747354 6530 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:12.830526 6530 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:34469 I0517 06:44:12.831404 6530 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:44:12.831506 6530 webserver.cc:184] Document root disabled I0517 06:44:12.832684 6530 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:48876/ I0517 06:44:12.833356 6530 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:44:12.837050 6530 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:12.839996 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 6530 I0517 06:44:12.841296 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:39669 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:12.856019 6651 heartbeater.cc:346] Connected to a master server at 127.4.96.62:39669 I0517 06:44:12.856243 6651 heartbeater.cc:426] Registering TS with master... I0517 06:44:12.856657 6651 heartbeater.cc:475] Master 127.4.96.62:39669 requested a full tablet report, sending... I0517 06:44:12.856756 6402 heartbeater.cc:467] Master 127.4.96.62:39669 was elected leader, sending a full tablet report... I0517 06:44:12.858366 6236 ts_manager.cc:81] Registered new tserver with Master: c95723aafe594cc999bfe59b9aee2b1e (127.4.96.3:34469) I0517 06:44:12.859751 6236 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:43137 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:12.945747 6654 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:12.946084 6654 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:12.953900 6654 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:12.954234 6654 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:44:12.966629 6654 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:39669 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:12.974757 6654 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:12.976563 6654 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 125319us I0517 06:44:12.976989 6654 fs_manager.cc:260] Metadata directory not provided I0517 06:44:12.977111 6654 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:12.977313 6654 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:44:12.977424 6654 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:12.984853 6654 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "e4842316d7f843469d36409ae004670c" format_stamp: "Formatted at 2018-05-17 06:44:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:12.985520 6654 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "e4842316d7f843469d36409ae004670c" format_stamp: "Formatted at 2018-05-17 06:44:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:12.994810 6654 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:44:12.995874 6654 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:12.996266 6654 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:13.002571 6654 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:44:13.002684 6654 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "e4842316d7f843469d36409ae004670c" format_stamp: "Formatted at 2018-05-17 06:44:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:13.002888 6654 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:13.174319 6654 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:13.176646 6654 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:13.177594 6654 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:13.270551 6654 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:35474 I0517 06:44:13.271446 6654 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:44:13.271553 6654 webserver.cc:184] Document root disabled I0517 06:44:13.272786 6654 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:57169/ I0517 06:44:13.273488 6654 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:44:13.276005 6654 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:13.276955 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 6654 I0517 06:44:13.278208 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.4.96.5:0 --local_ip_for_outbound_sockets=127.4.96.5 --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:39669 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:13.296645 6775 heartbeater.cc:346] Connected to a master server at 127.4.96.62:39669 I0517 06:44:13.296888 6775 heartbeater.cc:426] Registering TS with master... I0517 06:44:13.297315 6775 heartbeater.cc:475] Master 127.4.96.62:39669 requested a full tablet report, sending... I0517 06:44:13.298728 6236 ts_manager.cc:81] Registered new tserver with Master: e4842316d7f843469d36409ae004670c (127.4.96.4:35474) I0517 06:44:13.299818 6236 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:35432 I0517 06:44:13.373195 6527 heartbeater.cc:467] Master 127.4.96.62:39669 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:13.384343 6778 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:13.384654 6778 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:13.392498 6778 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:13.392804 6778 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.5 I0517 06:44:13.405079 6778 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:39669 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.5 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:13.413009 6778 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:13.414752 6778 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 125819us I0517 06:44:13.415150 6778 fs_manager.cc:260] Metadata directory not provided I0517 06:44:13.415256 6778 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:13.415432 6778 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0517 06:44:13.415534 6778 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:13.423193 6778 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "a668d75502e54c95abf2e1eceee63ab6" format_stamp: "Formatted at 2018-05-17 06:44:13 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:13.423841 6778 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "a668d75502e54c95abf2e1eceee63ab6" format_stamp: "Formatted at 2018-05-17 06:44:13 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:13.433058 6778 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:44:13.434177 6778 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:13.434556 6778 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:13.440789 6778 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:44:13.440896 6778 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal uuid: "a668d75502e54c95abf2e1eceee63ab6" format_stamp: "Formatted at 2018-05-17 06:44:13 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:13.441076 6778 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:13.592043 6778 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:13.594341 6778 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:13.595306 6778 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:13.680112 6778 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.5:57208 I0517 06:44:13.680977 6778 webserver.cc:173] Starting webserver on 127.4.96.5:0 I0517 06:44:13.681073 6778 webserver.cc:184] Document root disabled I0517 06:44:13.682255 6778 webserver.cc:311] Webserver started. Bound to: http://127.4.96.5:43549/ I0517 06:44:13.682931 6778 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb I0517 06:44:13.688351 6778 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:13.692962 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 6778 I0517 06:44:13.703902 6899 heartbeater.cc:346] Connected to a master server at 127.4.96.62:39669 I0517 06:44:13.704118 6899 heartbeater.cc:426] Registering TS with master... I0517 06:44:13.704526 6899 heartbeater.cc:475] Master 127.4.96.62:39669 requested a full tablet report, sending... I0517 06:44:13.705826 6237 ts_manager.cc:81] Registered new tserver with Master: a668d75502e54c95abf2e1eceee63ab6 (127.4.96.5:57208) I0517 06:44:13.707448 6237 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.5:43172 I0517 06:44:13.707854 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:44:13.725845 6237 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35376: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:44:13.769748 6712 tablet_service.cc:759] Processing CreateTablet for tablet e87d26c881e34c10ad80ac7338847fd0 (table=TestTable [id=1703e92e40bd4cfdbac9a9d2ebfd525a]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:13.770689 6712 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e87d26c881e34c10ad80ac7338847fd0. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:13.773376 6588 tablet_service.cc:759] Processing CreateTablet for tablet e87d26c881e34c10ad80ac7338847fd0 (table=TestTable [id=1703e92e40bd4cfdbac9a9d2ebfd525a]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:13.774042 6588 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e87d26c881e34c10ad80ac7338847fd0. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:13.775403 6339 tablet_service.cc:759] Processing CreateTablet for tablet e87d26c881e34c10ad80ac7338847fd0 (table=TestTable [id=1703e92e40bd4cfdbac9a9d2ebfd525a]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:13.776226 6339 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e87d26c881e34c10ad80ac7338847fd0. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:13.782629 6712 ts_tablet_manager.cc:1173] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:13.786684 6919 ts_tablet_manager.cc:1028] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c: Bootstrapping tablet I0517 06:44:13.787165 6919 tablet_bootstrap.cc:436] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c: Bootstrap starting. I0517 06:44:13.789786 6588 ts_tablet_manager.cc:1173] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:13.790153 6919 tablet_bootstrap.cc:587] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c: No blocks or log segments found. Creating new log. I0517 06:44:13.791568 6919 log.cc:520] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c: Log is configured to *not* fsync() on all Append() calls I0517 06:44:13.791573 6920 ts_tablet_manager.cc:1028] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e: Bootstrapping tablet I0517 06:44:13.791961 6339 ts_tablet_manager.cc:1173] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:13.792038 6920 tablet_bootstrap.cc:436] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e: Bootstrap starting. I0517 06:44:13.796613 6920 tablet_bootstrap.cc:587] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e: No blocks or log segments found. Creating new log. I0517 06:44:13.798086 6920 log.cc:520] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e: Log is configured to *not* fsync() on all Append() calls I0517 06:44:13.799664 6919 tablet_bootstrap.cc:436] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c: No bootstrap required, opened a new log I0517 06:44:13.799640 6922 ts_tablet_manager.cc:1028] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Bootstrapping tablet I0517 06:44:13.800034 6919 ts_tablet_manager.cc:1045] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.012s I0517 06:44:13.800071 6922 tablet_bootstrap.cc:436] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Bootstrap starting. I0517 06:44:13.802623 6922 tablet_bootstrap.cc:587] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: No blocks or log segments found. Creating new log. I0517 06:44:13.803064 6920 tablet_bootstrap.cc:436] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e: No bootstrap required, opened a new log I0517 06:44:13.803433 6920 ts_tablet_manager.cc:1045] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s I0517 06:44:13.803894 6922 log.cc:520] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Log is configured to *not* fsync() on all Append() calls I0517 06:44:13.815384 6919 raft_consensus.cc:303] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 } } peers { permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 } } peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } I0517 06:44:13.816119 6919 raft_consensus.cc:329] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:13.816318 6919 raft_consensus.cc:604] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4842316d7f843469d36409ae004670c, State: Initialized, Role: FOLLOWER I0517 06:44:13.816668 6922 tablet_bootstrap.cc:436] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: No bootstrap required, opened a new log I0517 06:44:13.816936 6922 ts_tablet_manager.cc:1045] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s I0517 06:44:13.816923 6919 consensus_queue.cc:227] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 } } peers { permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 } } peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } I0517 06:44:13.818958 6775 heartbeater.cc:467] Master 127.4.96.62:39669 was elected leader, sending a full tablet report... I0517 06:44:13.820542 6919 ts_tablet_manager.cc:1073] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c: Time spent starting tablet: real 0.020s user 0.016s sys 0.000s I0517 06:44:13.824859 6920 raft_consensus.cc:303] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 } } peers { permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 } } peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } I0517 06:44:13.825389 6920 raft_consensus.cc:329] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:13.825531 6920 raft_consensus.cc:604] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95723aafe594cc999bfe59b9aee2b1e, State: Initialized, Role: FOLLOWER I0517 06:44:13.825985 6920 consensus_queue.cc:227] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 } } peers { permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 } } peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } I0517 06:44:13.827702 6651 heartbeater.cc:467] Master 127.4.96.62:39669 was elected leader, sending a full tablet report... I0517 06:44:13.828492 6922 raft_consensus.cc:303] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 } } peers { permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 } } peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } I0517 06:44:13.829016 6922 raft_consensus.cc:329] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:13.829156 6922 raft_consensus.cc:604] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9766287bc0d4d29927a018255e773e1, State: Initialized, Role: FOLLOWER I0517 06:44:13.829582 6922 consensus_queue.cc:227] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 } } peers { permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 } } peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } W0517 06:44:13.836251 6652 tablet.cc:1678] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:13.844374 6922 ts_tablet_manager.cc:1073] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Time spent starting tablet: real 0.027s user 0.016s sys 0.008s I0517 06:44:13.828075 6920 ts_tablet_manager.cc:1073] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e: Time spent starting tablet: real 0.024s user 0.016s sys 0.000s I0517 06:44:13.890041 6927 raft_consensus.cc:435] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:44:13.890244 6927 raft_consensus.cc:2704] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Snoozing failure detection for 1.607s (starting election) I0517 06:44:13.890446 6927 raft_consensus.cc:457] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 } } peers { permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 } } peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } I0517 06:44:13.891321 6927 leader_election.cc:243] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c95723aafe594cc999bfe59b9aee2b1e (127.4.96.3:34469) I0517 06:44:13.891721 6927 leader_election.cc:243] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e4842316d7f843469d36409ae004670c (127.4.96.4:35474) I0517 06:44:13.909162 6732 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e87d26c881e34c10ad80ac7338847fd0" candidate_uuid: "f9766287bc0d4d29927a018255e773e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4842316d7f843469d36409ae004670c" is_pre_election: true I0517 06:44:13.909585 6732 raft_consensus.cc:2704] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c: Snoozing failure detection for 1.848s (vote granted) I0517 06:44:13.909788 6732 raft_consensus.cc:2219] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9766287bc0d4d29927a018255e773e1 in term 0. I0517 06:44:13.910717 6291 leader_election.cc:379] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [CANDIDATE]: Term 1 pre-election: Vote granted by peer e4842316d7f843469d36409ae004670c (127.4.96.4:35474) I0517 06:44:13.910902 6291 leader_election.cc:270] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:44:13.911280 6927 raft_consensus.cc:2704] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Snoozing failure detection for 1.793s (election complete) I0517 06:44:13.911242 6608 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e87d26c881e34c10ad80ac7338847fd0" candidate_uuid: "f9766287bc0d4d29927a018255e773e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95723aafe594cc999bfe59b9aee2b1e" is_pre_election: true I0517 06:44:13.911506 6927 raft_consensus.cc:2511] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:44:13.911689 6927 raft_consensus.cc:435] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:44:13.911793 6608 raft_consensus.cc:2704] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e: Snoozing failure detection for 1.520s (vote granted) I0517 06:44:13.911880 6927 raft_consensus.cc:2704] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Snoozing failure detection for 1.730s (starting election) I0517 06:44:13.912065 6927 raft_consensus.cc:2756] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:13.912041 6608 raft_consensus.cc:2219] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9766287bc0d4d29927a018255e773e1 in term 0. I0517 06:44:13.915478 6927 raft_consensus.cc:457] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 } } peers { permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 } } peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } I0517 06:44:13.916103 6927 leader_election.cc:243] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [CANDIDATE]: Term 1 election: Requesting vote from peer c95723aafe594cc999bfe59b9aee2b1e (127.4.96.3:34469) I0517 06:44:13.916373 6927 leader_election.cc:243] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [CANDIDATE]: Term 1 election: Requesting vote from peer e4842316d7f843469d36409ae004670c (127.4.96.4:35474) I0517 06:44:13.917356 6732 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e87d26c881e34c10ad80ac7338847fd0" candidate_uuid: "f9766287bc0d4d29927a018255e773e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4842316d7f843469d36409ae004670c" I0517 06:44:13.917747 6732 raft_consensus.cc:2756] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:13.918000 6732 raft_consensus.cc:2704] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c: Snoozing failure detection for 2.098s (vote granted) I0517 06:44:13.921988 6732 raft_consensus.cc:2219] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9766287bc0d4d29927a018255e773e1 in term 1. I0517 06:44:13.922740 6291 leader_election.cc:379] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [CANDIDATE]: Term 1 election: Vote granted by peer e4842316d7f843469d36409ae004670c (127.4.96.4:35474) I0517 06:44:13.922976 6291 leader_election.cc:270] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:44:13.923380 6927 raft_consensus.cc:2704] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Snoozing failure detection for 1.572s (election complete) I0517 06:44:13.923563 6927 raft_consensus.cc:2511] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:44:13.912989 6290 leader_election.cc:379] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [CANDIDATE]: Term 1 pre-election: Vote granted by peer c95723aafe594cc999bfe59b9aee2b1e (127.4.96.3:34469) W0517 06:44:13.969969 6290 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.96.3:34469, user_credentials={real_user=slave}} blocked reactor thread for 57064.6us I0517 06:44:13.970204 6927 raft_consensus.cc:571] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 1 LEADER]: Becoming Leader. State: Replica: f9766287bc0d4d29927a018255e773e1, State: Running, Role: LEADER I0517 06:44:13.970680 6927 consensus_queue.cc:204] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 } } peers { permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 } } peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } I0517 06:44:13.971216 6608 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e87d26c881e34c10ad80ac7338847fd0" candidate_uuid: "f9766287bc0d4d29927a018255e773e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95723aafe594cc999bfe59b9aee2b1e" I0517 06:44:13.971532 6608 raft_consensus.cc:2756] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:13.971707 6608 raft_consensus.cc:2704] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e: Snoozing failure detection for 2.008s (vote granted) I0517 06:44:13.975328 6608 raft_consensus.cc:2219] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9766287bc0d4d29927a018255e773e1 in term 1. I0517 06:44:13.976229 6290 leader_election.cc:379] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [CANDIDATE]: Term 1 election: Vote granted by peer c95723aafe594cc999bfe59b9aee2b1e (127.4.96.3:34469) I0517 06:44:13.976860 6236 catalog_manager.cc:3686] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 reported cstate change: term changed from 0 to 1, leader changed from to f9766287bc0d4d29927a018255e773e1 (127.4.96.1). New cstate: current_term: 1 leader_uuid: "f9766287bc0d4d29927a018255e773e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } health_report { overall_health: HEALTHY } } } I0517 06:44:14.025573 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:44:14.027557 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f9766287bc0d4d29927a018255e773e1 to finish bootstrapping I0517 06:44:14.037509 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b381de9177314b1294f8324c1ea8ef96 to finish bootstrapping I0517 06:44:14.048386 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c95723aafe594cc999bfe59b9aee2b1e to finish bootstrapping I0517 06:44:14.057471 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e4842316d7f843469d36409ae004670c to finish bootstrapping I0517 06:44:14.066628 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a668d75502e54c95abf2e1eceee63ab6 to finish bootstrapping W0517 06:44:14.074939 6403 tablet.cc:1678] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:14.077944 4480 kudu-admin-test.cc:477] Waiting for Master to see the current replicas... I0517 06:44:14.080255 4480 kudu-admin-test.cc:484] Tablet locations: tablet_id: "e87d26c881e34c10ad80ac7338847fd0" replicas { ts_info { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" rpc_addresses { host: "127.4.96.3" port: 34469 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "e4842316d7f843469d36409ae004670c" rpc_addresses { host: "127.4.96.4" port: 35474 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" rpc_addresses { host: "127.4.96.1" port: 33942 } } role: LEADER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } W0517 06:44:14.028139 6776 tablet.cc:1678] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:14.481315 6608 raft_consensus.cc:1057] T e87d26c881e34c10ad80ac7338847fd0 P c95723aafe594cc999bfe59b9aee2b1e [term 1 FOLLOWER]: Refusing update from remote peer f9766287bc0d4d29927a018255e773e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:14.482445 6936 consensus_queue.cc:963] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:14.493008 6732 raft_consensus.cc:1057] T e87d26c881e34c10ad80ac7338847fd0 P e4842316d7f843469d36409ae004670c [term 1 FOLLOWER]: Refusing update from remote peer f9766287bc0d4d29927a018255e773e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:14.494601 6940 consensus_queue.cc:963] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:14.499930 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 6654 W0517 06:44:14.516983 6219 connection.cc:512] client connection to 127.4.96.4:35474 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:35474): Connection reset by peer (error 104) W0517 06:44:14.517094 6208 connection.cc:512] client connection to 127.4.96.4:35474 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:35474): Connection reset by peer (error 104) W0517 06:44:14.518043 6217 connection.cc:512] server connection from 127.4.96.4:35432 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:35432): Connection reset by peer (error 104) W0517 06:44:14.518234 6291 connection.cc:512] client connection to 127.4.96.4:35474 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:35474): Connection reset by peer (error 104) W0517 06:44:14.518481 6291 consensus_peers.cc:434] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 -> Peer e4842316d7f843469d36409ae004670c (127.4.96.4:35474): Couldn't send request to peer e4842316d7f843469d36409ae004670c for tablet e87d26c881e34c10ad80ac7338847fd0. Status: Network error: failed to read from TLS socket (remote: 127.4.96.4:35474): Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:14.519400 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 6530 W0517 06:44:14.536175 6218 connection.cc:512] client connection to 127.4.96.3:34469 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:34469): Connection reset by peer (error 104) W0517 06:44:14.536195 6290 connection.cc:512] client connection to 127.4.96.3:34469 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:34469): Connection reset by peer (error 104) I0517 06:44:14.536401 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 6209 W0517 06:44:14.537189 6208 connection.cc:512] client connection to 127.4.96.3:34469 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:34469): Connection reset by peer (error 104) W0517 06:44:14.567142 6293 connection.cc:512] client connection to 127.4.96.62:39669 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:39669): Connection reset by peer (error 104) W0517 06:44:14.567173 6418 connection.cc:512] client connection to 127.4.96.62:39669 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:39669): Connection reset by peer (error 104) W0517 06:44:14.568358 6790 connection.cc:512] client connection to 127.4.96.62:39669 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:39669): Connection reset by peer (error 104) W0517 06:44:14.569177 6293 connection.cc:512] server connection from 127.0.0.1:36849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36849): Connection reset by peer (error 104) W0517 06:44:14.569212 6208 connection.cc:512] client connection to 127.4.96.62:39669 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:39669): Connection reset by peer (error 104) W0517 06:44:14.569324 6907 connection.cc:512] client connection to 127.4.96.62:39669 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:39669): Connection reset by peer (error 104) I0517 06:44:14.574144 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:39669 --webserver_interface=127.4.96.62 --webserver_port=51907 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:14.690415 6943 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:14.690740 6943 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:14.699615 6943 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:44:14.699741 6943 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:14.699864 6943 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:44:14.699998 6943 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:44:14.710242 6899 heartbeater.cc:467] Master 127.4.96.62:39669 was elected leader, sending a full tablet report... W0517 06:44:14.715785 6899 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:39669: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.96.62:39669: connect: Connection refused (error 111) I0517 06:44:14.716655 6943 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:39669 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=51907 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:14.724421 6943 master_main.cc:80] Initializing master server... I0517 06:44:14.725967 6943 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 126319us I0517 06:44:14.726702 6943 fs_manager.cc:260] Metadata directory not provided I0517 06:44:14.726836 6943 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:14.731946 6943 fs_manager.cc:397] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s I0517 06:44:14.732559 6943 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:14.732977 6943 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:14.739357 6943 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.008s sys 0.000s I0517 06:44:14.739472 6943 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "354b55e91977404abb71f4ca603fe80a" format_stamp: "Formatted at 2018-05-17 06:44:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:14.740581 6943 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0517 06:44:14.870867 6290 consensus_peers.cc:434] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 -> Peer c95723aafe594cc999bfe59b9aee2b1e (127.4.96.3:34469): Couldn't send request to peer c95723aafe594cc999bfe59b9aee2b1e for tablet e87d26c881e34c10ad80ac7338847fd0. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.3:34469: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:14.896818 6943 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:14.898519 6943 master_main.cc:83] Starting Master server... I0517 06:44:14.938633 6943 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:39669 I0517 06:44:14.939527 6943 webserver.cc:173] Starting webserver on 127.4.96.62:51907 I0517 06:44:14.939636 6943 webserver.cc:184] Document root disabled I0517 06:44:14.940876 6943 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:51907/ I0517 06:44:14.941700 6943 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:44:14.950562 7006 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: Bootstrap starting. I0517 06:44:14.951735 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 6943 I0517 06:44:14.952131 4480 kudu-admin-test.cc:503] Forcing unsafe config change on tserver f9766287bc0d4d29927a018255e773e1 W0517 06:44:14.960248 6291 consensus_peers.cc:434] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 -> Peer e4842316d7f843469d36409ae004670c (127.4.96.4:35474): Couldn't send request to peer e4842316d7f843469d36409ae004670c for tablet e87d26c881e34c10ad80ac7338847fd0. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.4:35474: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I0517 06:44:14.977229 7006 log.cc:520] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: Log is configured to *not* fsync() on all Append() calls I0517 06:44:14.988688 7006 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:14.989233 7006 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: Bootstrap complete. I0517 06:44:15.007102 7006 raft_consensus.cc:303] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } I0517 06:44:15.008697 7006 raft_consensus.cc:604] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354b55e91977404abb71f4ca603fe80a, State: Initialized, Role: FOLLOWER I0517 06:44:15.009165 7006 consensus_queue.cc:227] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } I0517 06:44:15.009510 7006 raft_consensus.cc:343] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:44:15.009658 7006 raft_consensus.cc:435] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:44:15.009805 7006 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: Snoozing failure detection for 2.249s (starting election) I0517 06:44:15.009958 7006 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 1 FOLLOWER]: Advancing to term 2 I0517 06:44:15.014333 7006 raft_consensus.cc:457] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } I0517 06:44:15.014753 7006 leader_election.cc:270] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0517 06:44:15.016425 7009 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: Snoozing failure detection for 1.858s (election complete) I0517 06:44:15.016703 7009 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 2 FOLLOWER]: Leader election won for term 2 I0517 06:44:15.026415 7009 raft_consensus.cc:571] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [term 2 LEADER]: Becoming Leader. State: Replica: 354b55e91977404abb71f4ca603fe80a, State: Running, Role: LEADER I0517 06:44:15.026834 7009 consensus_queue.cc:204] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } I0517 06:44:15.030236 7006 sys_catalog.cc:335] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "354b55e91977404abb71f4ca603fe80a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } } I0517 06:44:15.030658 7006 sys_catalog.cc:338] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: This master's current role is: LEADER I0517 06:44:15.032694 7006 sys_catalog.cc:422] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: configured and running, proceeding with master startup. W0517 06:44:15.077307 7007 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:44:15.032847 7013 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:44:15.104667 6943 master_main.cc:86] Master server successfully started. I0517 06:44:15.033766 7011 sys_catalog.cc:335] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 354b55e91977404abb71f4ca603fe80a. Latest consensus state: current_term: 2 leader_uuid: "354b55e91977404abb71f4ca603fe80a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } } I0517 06:44:15.115988 7011 sys_catalog.cc:338] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: This master's current role is: LEADER I0517 06:44:15.033450 7010 sys_catalog.cc:335] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "354b55e91977404abb71f4ca603fe80a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354b55e91977404abb71f4ca603fe80a" member_type: VOTER } } I0517 06:44:15.118562 7010 sys_catalog.cc:338] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a [sys.catalog]: This master's current role is: LEADER I0517 06:44:15.122740 7013 catalog_manager.cc:321] Loaded metadata for table TestTable [id=1703e92e40bd4cfdbac9a9d2ebfd525a] I0517 06:44:15.125641 7013 catalog_manager.cc:375] Loaded metadata for tablet e87d26c881e34c10ad80ac7338847fd0 (table TestTable [id=1703e92e40bd4cfdbac9a9d2ebfd525a]) I0517 06:44:15.126240 7013 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:44:15.131489 7013 catalog_manager.cc:1021] Loading token signing keys... I0517 06:44:15.133622 7013 catalog_manager.cc:3973] T 00000000000000000000000000000000 P 354b55e91977404abb71f4ca603fe80a: Loaded TSK: 0 I0517 06:44:15.325562 6359 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "f9766287bc0d4d29927a018255e773e1" tablet_id: "e87d26c881e34c10ad80ac7338847fd0" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" } } from {username='slave'} at 127.0.0.1:36857 W0517 06:44:15.326467 6359 raft_consensus.cc:1990] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 1 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 } } peers { permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 } } peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } }NEW CONFIG: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } unsafe_config_change: true I0517 06:44:15.328135 6359 raft_consensus.cc:2751] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 1 LEADER]: Stepping down as leader of term 1 I0517 06:44:15.328343 6359 raft_consensus.cc:604] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9766287bc0d4d29927a018255e773e1, State: Running, Role: LEADER I0517 06:44:15.328821 6359 consensus_queue.cc:227] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 } } peers { permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 } } peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } I0517 06:44:15.329486 6359 consensus_peers.cc:456] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 -> Peer e4842316d7f843469d36409ae004670c (127.4.96.4:35474): Closing peer: e4842316d7f843469d36409ae004670c I0517 06:44:15.329684 6359 consensus_peers.cc:456] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 -> Peer c95723aafe594cc999bfe59b9aee2b1e (127.4.96.3:34469): Closing peer: c95723aafe594cc999bfe59b9aee2b1e I0517 06:44:15.330126 6359 raft_consensus.cc:2756] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:44:15.352344 6973 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "f9766287bc0d4d29927a018255e773e1" instance_seqno: 1526539451795272) as {username='slave'} at 127.4.96.1:32827; Asking this server to re-register. I0517 06:44:15.353647 6402 heartbeater.cc:426] Registering TS with master... I0517 06:44:15.353898 6402 heartbeater.cc:475] Master 127.4.96.62:39669 requested a full tablet report, sending... I0517 06:44:15.355391 6973 ts_manager.cc:81] Registered new tserver with Master: f9766287bc0d4d29927a018255e773e1 (127.4.96.1:33942) I0517 06:44:15.387966 6973 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" instance_seqno: 1526539452313691) as {username='slave'} at 127.4.96.2:49745; Asking this server to re-register. I0517 06:44:15.388845 6527 heartbeater.cc:426] Registering TS with master... I0517 06:44:15.389173 6527 heartbeater.cc:475] Master 127.4.96.62:39669 requested a full tablet report, sending... I0517 06:44:15.390278 6973 ts_manager.cc:81] Registered new tserver with Master: b381de9177314b1294f8324c1ea8ef96 (127.4.96.2:35758) I0517 06:44:15.728672 6899 heartbeater.cc:346] Connected to a master server at 127.4.96.62:39669 I0517 06:44:15.728932 6899 heartbeater.cc:467] Master 127.4.96.62:39669 was elected leader, sending a full tablet report... I0517 06:44:15.730270 6973 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" instance_seqno: 1526539453662927) as {username='slave'} at 127.4.96.5:39457; Asking this server to re-register. I0517 06:44:15.731416 6899 heartbeater.cc:426] Registering TS with master... I0517 06:44:15.731652 6899 heartbeater.cc:475] Master 127.4.96.62:39669 requested a full tablet report, sending... I0517 06:44:15.732825 6973 ts_manager.cc:81] Registered new tserver with Master: a668d75502e54c95abf2e1eceee63ab6 (127.4.96.5:57208) I0517 06:44:16.836079 7037 raft_consensus.cc:435] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0517 06:44:16.836294 7037 raft_consensus.cc:2704] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Snoozing failure detection for 1.646s (starting election) I0517 06:44:16.836480 7037 raft_consensus.cc:457] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } unsafe_config_change: true I0517 06:44:16.836889 7037 leader_election.cc:270] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0517 06:44:16.837915 7037 raft_consensus.cc:2704] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Snoozing failure detection for 1.682s (election complete) I0517 06:44:16.838099 7037 raft_consensus.cc:2511] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0517 06:44:16.838230 7037 raft_consensus.cc:435] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0517 06:44:16.838373 7037 raft_consensus.cc:2704] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Snoozing failure detection for 1.710s (starting election) I0517 06:44:16.838505 7037 raft_consensus.cc:2756] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 2 FOLLOWER]: Advancing to term 3 I0517 06:44:16.841251 7037 raft_consensus.cc:457] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } unsafe_config_change: true I0517 06:44:16.841608 7037 leader_election.cc:270] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0517 06:44:16.842061 7038 raft_consensus.cc:2704] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Snoozing failure detection for 1.706s (election complete) I0517 06:44:16.842237 7038 raft_consensus.cc:2511] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 3 FOLLOWER]: Leader election won for term 3 I0517 06:44:16.842448 7038 raft_consensus.cc:571] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 3 LEADER]: Becoming Leader. State: Replica: f9766287bc0d4d29927a018255e773e1, State: Running, Role: LEADER I0517 06:44:16.842710 7038 consensus_queue.cc:204] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } unsafe_config_change: true I0517 06:44:16.844866 7038 raft_consensus.cc:2658] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 3 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER c95723aafe594cc999bfe59b9aee2b1e (127.4.96.3) evicted, VOTER e4842316d7f843469d36409ae004670c (127.4.96.4) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } unsafe_config_change: true } I0517 06:44:16.846158 6973 catalog_manager.cc:3686] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 reported cstate change: term changed from 1 to 3, now has a pending config: VOTER f9766287bc0d4d29927a018255e773e1 (127.4.96.1). New cstate: current_term: 3 leader_uuid: "f9766287bc0d4d29927a018255e773e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95723aafe594cc999bfe59b9aee2b1e" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 34469 } } peers { permanent_uuid: "e4842316d7f843469d36409ae004670c" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 35474 } } peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } unsafe_config_change: true } I0517 06:44:16.864070 6973 catalog_manager.cc:3686] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 reported cstate change: config changed from index -1 to 2, VOTER c95723aafe594cc999bfe59b9aee2b1e (127.4.96.3) evicted, VOTER e4842316d7f843469d36409ae004670c (127.4.96.4) evicted, no longer has a pending config: VOTER f9766287bc0d4d29927a018255e773e1 (127.4.96.1). New cstate: current_term: 3 leader_uuid: "f9766287bc0d4d29927a018255e773e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:44:16.867478 6973 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e with a delay of 26 ms (attempt = 1) W0517 06:44:16.867650 6973 catalog_manager.cc:3845] Failed to send e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e: Not found: failed to reset TS proxy: Could not find TS for UUID c95723aafe594cc999bfe59b9aee2b1e I0517 06:44:16.867800 6973 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c with a delay of 21 ms (attempt = 1) W0517 06:44:16.867920 6973 catalog_manager.cc:3845] Failed to send e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c: Not found: failed to reset TS proxy: Could not find TS for UUID e4842316d7f843469d36409ae004670c I0517 06:44:16.868304 6973 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet e87d26c881e34c10ad80ac7338847fd0 (attempt 1) I0517 06:44:16.878463 6359 consensus_queue.cc:204] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:16.879901 7037 raft_consensus.cc:2658] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER b381de9177314b1294f8324c1ea8ef96 (127.4.96.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:16.885421 6953 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e87d26c881e34c10ad80ac7338847fd0 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0517 06:44:16.889582 6954 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c with a delay of 50 ms (attempt = 2) W0517 06:44:16.889793 6954 catalog_manager.cc:2810] Async tablet task e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c failed: Not found: failed to reset TS proxy: Could not find TS for UUID e4842316d7f843469d36409ae004670c I0517 06:44:16.893467 6973 catalog_manager.cc:3686] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 reported cstate change: config changed from index 2 to 4, NON_VOTER b381de9177314b1294f8324c1ea8ef96 (127.4.96.2) added. New cstate: current_term: 3 leader_uuid: "f9766287bc0d4d29927a018255e773e1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0517 06:44:16.894222 6952 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e with a delay of 43 ms (attempt = 2) W0517 06:44:16.894465 6952 catalog_manager.cc:2810] Async tablet task e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e failed: Not found: failed to reset TS proxy: Could not find TS for UUID c95723aafe594cc999bfe59b9aee2b1e I0517 06:44:16.897684 6973 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet e87d26c881e34c10ad80ac7338847fd0 (attempt 1) I0517 06:44:16.899387 6359 consensus_peers.cc:456] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 -> Peer b381de9177314b1294f8324c1ea8ef96 (127.4.96.2:35758): Closing peer: b381de9177314b1294f8324c1ea8ef96 I0517 06:44:16.899761 6359 consensus_queue.cc:204] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: true } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:16.903568 7048 raft_consensus.cc:2658] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, NON_VOTER a668d75502e54c95abf2e1eceee63ab6 (127.4.96.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: true } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: true } } unsafe_config_change: true } W0517 06:44:16.907625 6293 consensus_peers.cc:434] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 -> Peer b381de9177314b1294f8324c1ea8ef96 (127.4.96.2:35758): Couldn't send request to peer b381de9177314b1294f8324c1ea8ef96 for tablet e87d26c881e34c10ad80ac7338847fd0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e87d26c881e34c10ad80ac7338847fd0. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:16.909420 6953 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e87d26c881e34c10ad80ac7338847fd0 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0517 06:44:16.910701 6973 catalog_manager.cc:3686] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 reported cstate change: config changed from index 4 to 5, NON_VOTER a668d75502e54c95abf2e1eceee63ab6 (127.4.96.5) added. New cstate: current_term: 3 leader_uuid: "f9766287bc0d4d29927a018255e773e1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } W0517 06:44:16.919703 6293 consensus_peers.cc:434] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 -> Peer a668d75502e54c95abf2e1eceee63ab6 (127.4.96.5:57208): Couldn't send request to peer a668d75502e54c95abf2e1eceee63ab6 for tablet e87d26c881e34c10ad80ac7338847fd0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e87d26c881e34c10ad80ac7338847fd0. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:16.937937 6952 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e with a delay of 106 ms (attempt = 3) W0517 06:44:16.938093 6952 catalog_manager.cc:2810] Async tablet task e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e failed: Not found: failed to reset TS proxy: Could not find TS for UUID c95723aafe594cc999bfe59b9aee2b1e I0517 06:44:16.940208 6954 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c with a delay of 96 ms (attempt = 3) W0517 06:44:16.940387 6954 catalog_manager.cc:2810] Async tablet task e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c failed: Not found: failed to reset TS proxy: Could not find TS for UUID e4842316d7f843469d36409ae004670c I0517 06:44:17.037022 6954 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c with a delay of 174 ms (attempt = 4) W0517 06:44:17.037178 6954 catalog_manager.cc:2810] Async tablet task e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c failed: Not found: failed to reset TS proxy: Could not find TS for UUID e4842316d7f843469d36409ae004670c I0517 06:44:17.044708 6952 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e with a delay of 157 ms (attempt = 4) W0517 06:44:17.044891 6952 catalog_manager.cc:2810] Async tablet task e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e failed: Not found: failed to reset TS proxy: Could not find TS for UUID c95723aafe594cc999bfe59b9aee2b1e I0517 06:44:17.202744 6952 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e with a delay of 277 ms (attempt = 5) W0517 06:44:17.202978 6952 catalog_manager.cc:2810] Async tablet task e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e failed: Not found: failed to reset TS proxy: Could not find TS for UUID c95723aafe594cc999bfe59b9aee2b1e I0517 06:44:17.211977 6954 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c with a delay of 276 ms (attempt = 5) W0517 06:44:17.212167 6954 catalog_manager.cc:2810] Async tablet task e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c failed: Not found: failed to reset TS proxy: Could not find TS for UUID e4842316d7f843469d36409ae004670c I0517 06:44:17.289590 7050 ts_tablet_manager.cc:669] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: Initiating tablet copy from peer f9766287bc0d4d29927a018255e773e1 (127.4.96.1:33942) I0517 06:44:17.289866 7050 tablet_copy_client.cc:239] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.1:33942 I0517 06:44:17.299371 6379 tablet_copy_service.cc:133] P f9766287bc0d4d29927a018255e773e1: Received BeginTabletCopySession request for tablet e87d26c881e34c10ad80ac7338847fd0 from peer b381de9177314b1294f8324c1ea8ef96 ({username='slave'} at 127.4.96.2:39578) I0517 06:44:17.299604 6379 tablet_copy_service.cc:154] P f9766287bc0d4d29927a018255e773e1: Beginning new tablet copy session on tablet e87d26c881e34c10ad80ac7338847fd0 from peer b381de9177314b1294f8324c1ea8ef96 at {username='slave'} at 127.4.96.2:39578: session id = b381de9177314b1294f8324c1ea8ef96-e87d26c881e34c10ad80ac7338847fd0 I0517 06:44:17.302903 6379 tablet_copy_source_session.cc:202] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:44:17.305508 7050 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e87d26c881e34c10ad80ac7338847fd0. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:17.316164 7050 ts_tablet_manager.cc:1173] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:44:17.316504 7050 tablet_copy_client.cc:565] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: tablet copy: Starting download of 0 data blocks... I0517 06:44:17.316733 7050 tablet_copy_client.cc:528] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: tablet copy: Starting download of 1 WAL segments... I0517 06:44:17.318729 7050 tablet_copy_client.cc:404] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:44:17.321444 7050 ts_tablet_manager.cc:1028] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: Bootstrapping tablet I0517 06:44:17.321765 7050 tablet_bootstrap.cc:436] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: Bootstrap starting. I0517 06:44:17.326792 7050 log.cc:520] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: Log is configured to *not* fsync() on all Append() calls I0517 06:44:17.332136 7050 tablet_bootstrap.cc:436] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:17.332594 7050 tablet_bootstrap.cc:436] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: Bootstrap complete. I0517 06:44:17.332903 7050 ts_tablet_manager.cc:1045] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: Time spent bootstrapping tablet: real 0.011s user 0.012s sys 0.000s I0517 06:44:17.342078 7050 raft_consensus.cc:303] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: true } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:17.342656 7050 raft_consensus.cc:604] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: b381de9177314b1294f8324c1ea8ef96, State: Initialized, Role: LEARNER I0517 06:44:17.342967 7050 consensus_queue.cc:227] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: true } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:17.344833 7050 ts_tablet_manager.cc:1073] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s I0517 06:44:17.345738 6379 tablet_copy_service.cc:323] P f9766287bc0d4d29927a018255e773e1: Request end of tablet copy session b381de9177314b1294f8324c1ea8ef96-e87d26c881e34c10ad80ac7338847fd0 received from {username='slave'} at 127.4.96.2:39578 I0517 06:44:17.345911 6379 tablet_copy_service.cc:411] P f9766287bc0d4d29927a018255e773e1: Ending tablet copy session b381de9177314b1294f8324c1ea8ef96-e87d26c881e34c10ad80ac7338847fd0 on tablet e87d26c881e34c10ad80ac7338847fd0 with peer b381de9177314b1294f8324c1ea8ef96 I0517 06:44:17.481220 6952 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e with a delay of 546 ms (attempt = 6) W0517 06:44:17.481405 6952 catalog_manager.cc:2810] Async tablet task e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e failed: Not found: failed to reset TS proxy: Could not find TS for UUID c95723aafe594cc999bfe59b9aee2b1e I0517 06:44:17.489069 7054 ts_tablet_manager.cc:669] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: Initiating tablet copy from peer f9766287bc0d4d29927a018255e773e1 (127.4.96.1:33942) I0517 06:44:17.489274 6954 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c with a delay of 534 ms (attempt = 6) I0517 06:44:17.489419 7054 tablet_copy_client.cc:239] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.1:33942 W0517 06:44:17.489503 6954 catalog_manager.cc:2810] Async tablet task e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c failed: Not found: failed to reset TS proxy: Could not find TS for UUID e4842316d7f843469d36409ae004670c I0517 06:44:17.498260 6379 tablet_copy_service.cc:133] P f9766287bc0d4d29927a018255e773e1: Received BeginTabletCopySession request for tablet e87d26c881e34c10ad80ac7338847fd0 from peer a668d75502e54c95abf2e1eceee63ab6 ({username='slave'} at 127.4.96.5:39441) I0517 06:44:17.498488 6379 tablet_copy_service.cc:154] P f9766287bc0d4d29927a018255e773e1: Beginning new tablet copy session on tablet e87d26c881e34c10ad80ac7338847fd0 from peer a668d75502e54c95abf2e1eceee63ab6 at {username='slave'} at 127.4.96.5:39441: session id = a668d75502e54c95abf2e1eceee63ab6-e87d26c881e34c10ad80ac7338847fd0 I0517 06:44:17.501395 6379 tablet_copy_source_session.cc:202] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:44:17.503818 7054 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e87d26c881e34c10ad80ac7338847fd0. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:17.514654 7054 ts_tablet_manager.cc:1173] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:44:17.515002 7054 tablet_copy_client.cc:565] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: tablet copy: Starting download of 0 data blocks... I0517 06:44:17.515209 7054 tablet_copy_client.cc:528] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: tablet copy: Starting download of 1 WAL segments... I0517 06:44:17.516942 7054 tablet_copy_client.cc:404] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:44:17.519549 7054 ts_tablet_manager.cc:1028] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: Bootstrapping tablet I0517 06:44:17.519876 7054 tablet_bootstrap.cc:436] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: Bootstrap starting. I0517 06:44:17.524837 7054 log.cc:520] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: Log is configured to *not* fsync() on all Append() calls I0517 06:44:17.530222 7054 tablet_bootstrap.cc:436] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:17.530668 7054 tablet_bootstrap.cc:436] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: Bootstrap complete. I0517 06:44:17.530975 7054 ts_tablet_manager.cc:1045] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.004s I0517 06:44:17.540246 7054 raft_consensus.cc:303] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: true } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:17.540977 7054 raft_consensus.cc:604] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: a668d75502e54c95abf2e1eceee63ab6, State: Initialized, Role: LEARNER I0517 06:44:17.541345 7054 consensus_queue.cc:227] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: true } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:17.543471 7054 ts_tablet_manager.cc:1073] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: Time spent starting tablet: real 0.012s user 0.016s sys 0.000s I0517 06:44:17.544430 6379 tablet_copy_service.cc:323] P f9766287bc0d4d29927a018255e773e1: Request end of tablet copy session a668d75502e54c95abf2e1eceee63ab6-e87d26c881e34c10ad80ac7338847fd0 received from {username='slave'} at 127.4.96.5:39441 I0517 06:44:17.544602 6379 tablet_copy_service.cc:411] P f9766287bc0d4d29927a018255e773e1: Ending tablet copy session a668d75502e54c95abf2e1eceee63ab6-e87d26c881e34c10ad80ac7338847fd0 on tablet e87d26c881e34c10ad80ac7338847fd0 with peer a668d75502e54c95abf2e1eceee63ab6 W0517 06:44:17.592800 6528 tablet.cc:1678] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:44:17.690513 6900 tablet.cc:1678] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:17.829429 6484 raft_consensus.cc:1002] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.4->[3.5-3.5] Dedup: 3.5->[] I0517 06:44:18.024245 6954 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c with a delay of 1051 ms (attempt = 7) W0517 06:44:18.024442 6954 catalog_manager.cc:2810] Async tablet task e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=e4842316d7f843469d36409ae004670c failed: Not found: failed to reset TS proxy: Could not find TS for UUID e4842316d7f843469d36409ae004670c I0517 06:44:18.027902 6952 catalog_manager.cc:2791] Scheduling retry of e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e with a delay of 1058 ms (attempt = 7) W0517 06:44:18.028066 6952 catalog_manager.cc:2810] Async tablet task e87d26c881e34c10ad80ac7338847fd0 Delete Tablet RPC for TS=c95723aafe594cc999bfe59b9aee2b1e failed: Not found: failed to reset TS proxy: Could not find TS for UUID c95723aafe594cc999bfe59b9aee2b1e I0517 06:44:18.067286 6856 raft_consensus.cc:1002] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.4->[3.5-3.5] Dedup: 3.5->[] I0517 06:44:18.308781 7059 raft_consensus.cc:866] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: attempting to promote NON_VOTER b381de9177314b1294f8324c1ea8ef96 to VOTER I0517 06:44:18.309397 7059 consensus_peers.cc:456] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 -> Peer a668d75502e54c95abf2e1eceee63ab6 (127.4.96.5:57208): Closing peer: a668d75502e54c95abf2e1eceee63ab6 I0517 06:44:18.309536 7059 consensus_peers.cc:456] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 -> Peer b381de9177314b1294f8324c1ea8ef96 (127.4.96.2:35758): Closing peer: b381de9177314b1294f8324c1ea8ef96 I0517 06:44:18.309943 7059 consensus_queue.cc:204] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: false } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:18.313127 6484 raft_consensus.cc:1057] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96 [term 3 LEARNER]: Refusing update from remote peer f9766287bc0d4d29927a018255e773e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch) I0517 06:44:18.313124 6856 raft_consensus.cc:1057] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6 [term 3 LEARNER]: Refusing update from remote peer f9766287bc0d4d29927a018255e773e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch) I0517 06:44:18.314091 7048 consensus_queue.cc:963] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0517 06:44:18.314575 7058 consensus_queue.cc:963] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0517 06:44:18.319663 7048 raft_consensus.cc:2658] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, b381de9177314b1294f8324c1ea8ef96 (127.4.96.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: false } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:18.321568 6856 raft_consensus.cc:2658] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6 [term 3 LEARNER]: Committing config change with OpId 3.6: config changed from index 5 to 6, b381de9177314b1294f8324c1ea8ef96 (127.4.96.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: false } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:18.321789 6483 raft_consensus.cc:2658] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, b381de9177314b1294f8324c1ea8ef96 (127.4.96.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: false } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:18.325013 7058 raft_consensus.cc:866] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1: attempting to promote NON_VOTER a668d75502e54c95abf2e1eceee63ab6 to VOTER I0517 06:44:18.325753 7058 consensus_peers.cc:456] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 -> Peer a668d75502e54c95abf2e1eceee63ab6 (127.4.96.5:57208): Closing peer: a668d75502e54c95abf2e1eceee63ab6 I0517 06:44:18.325948 7058 consensus_peers.cc:456] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 -> Peer b381de9177314b1294f8324c1ea8ef96 (127.4.96.2:35758): Closing peer: b381de9177314b1294f8324c1ea8ef96 I0517 06:44:18.326272 7058 consensus_queue.cc:204] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: false } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: false } } unsafe_config_change: true I0517 06:44:18.326949 6972 catalog_manager.cc:3686] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 reported cstate change: config changed from index 5 to 6, b381de9177314b1294f8324c1ea8ef96 (127.4.96.2) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "f9766287bc0d4d29927a018255e773e1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: true } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:44:18.332037 6856 raft_consensus.cc:1057] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6 [term 3 LEARNER]: Refusing update from remote peer f9766287bc0d4d29927a018255e773e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:44:18.332504 6483 raft_consensus.cc:1057] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96 [term 3 FOLLOWER]: Refusing update from remote peer f9766287bc0d4d29927a018255e773e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:44:18.333695 7048 consensus_queue.cc:963] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:44:18.334244 7059 consensus_queue.cc:963] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:44:18.340168 7064 raft_consensus.cc:2658] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, a668d75502e54c95abf2e1eceee63ab6 (127.4.96.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: false } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:18.342160 6483 raft_consensus.cc:2658] T e87d26c881e34c10ad80ac7338847fd0 P b381de9177314b1294f8324c1ea8ef96 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, a668d75502e54c95abf2e1eceee63ab6 (127.4.96.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: false } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:18.343605 6856 raft_consensus.cc:2658] T e87d26c881e34c10ad80ac7338847fd0 P a668d75502e54c95abf2e1eceee63ab6 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, a668d75502e54c95abf2e1eceee63ab6 (127.4.96.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: false } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:18.346009 6971 catalog_manager.cc:3686] T e87d26c881e34c10ad80ac7338847fd0 P f9766287bc0d4d29927a018255e773e1 reported cstate change: config changed from index 6 to 7, a668d75502e54c95abf2e1eceee63ab6 (127.4.96.5) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "f9766287bc0d4d29927a018255e773e1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 33942 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 35758 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 57208 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:44:18.452546 4480 kudu-admin-test.cc:528] Waiting for Master to see new config... I0517 06:44:18.463021 4480 kudu-admin-test.cc:533] Tablet locations: tablet_id: "e87d26c881e34c10ad80ac7338847fd0" replicas { ts_info { permanent_uuid: "f9766287bc0d4d29927a018255e773e1" rpc_addresses { host: "127.4.96.1" port: 33942 } } role: LEADER } replicas { ts_info { permanent_uuid: "b381de9177314b1294f8324c1ea8ef96" rpc_addresses { host: "127.4.96.2" port: 35758 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "a668d75502e54c95abf2e1eceee63ab6" rpc_addresses { host: "127.4.96.5" port: 57208 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0517 06:44:18.464243 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 6271 W0517 06:44:18.486192 6953 connection.cc:512] client connection to 127.4.96.1:33942 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:33942): Connection reset by peer (error 104) W0517 06:44:18.486225 6415 connection.cc:512] client connection to 127.4.96.1:33942 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:33942): Connection reset by peer (error 104) W0517 06:44:18.487191 6954 connection.cc:512] server connection from 127.4.96.1:32827 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:32827): Connection reset by peer (error 104) W0517 06:44:18.487277 6414 connection.cc:512] server connection from 127.4.96.1:45378 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:45378): Connection reset by peer (error 104) W0517 06:44:18.487248 6208 connection.cc:512] client connection to 127.4.96.1:33942 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:33942): Connection reset by peer (error 104) W0517 06:44:18.488137 6205 connection.cc:512] client connection to 127.4.96.1:33942 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:33942): Connection reset by peer (error 104) W0517 06:44:18.488180 6787 connection.cc:512] server connection from 127.4.96.1:38932 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:38932): Connection reset by peer (error 104) W0517 06:44:18.488972 6787 connection.cc:512] client connection to 127.4.96.1:33942 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:33942): Connection reset by peer (error 104) I0517 06:44:18.489754 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 6406 W0517 06:44:18.508704 6207 connection.cc:512] client connection to 127.4.96.2:35758 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:35758): Connection reset by peer (error 104) W0517 06:44:18.508723 6953 connection.cc:512] server connection from 127.4.96.2:49745 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:49745): Connection reset by peer (error 104) I0517 06:44:18.508762 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 6778 W0517 06:44:18.509503 6208 connection.cc:512] client connection to 127.4.96.2:35758 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:35758): Connection reset by peer (error 104) W0517 06:44:18.533407 6208 connection.cc:512] client connection to 127.4.96.5:57208 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:57208): Connection reset by peer (error 104) W0517 06:44:18.533447 6953 connection.cc:512] server connection from 127.4.96.5:39457 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:39457): Connection reset by peer (error 104) I0517 06:44:18.534297 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 6943 W0517 06:44:18.561491 6208 connection.cc:512] client connection to 127.4.96.62:39669 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:39669): Connection reset by peer (error 104) [ OK ] AdminCliTest.TestUnsafeChangeConfigOnSingleLeader (7705 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes I0517 06:44:18.576352 4480 test_util.cc:200] Using random seed: 707395532 I0517 06:44:18.579404 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:44:18.579499 4480 ts_itest-base.cc:114] -------------- I0517 06:44:18.579574 4480 ts_itest-base.cc:115] 4 tablet servers I0517 06:44:18.579646 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:44:18.579721 4480 ts_itest-base.cc:117] -------------- I0517 06:44:18.581981 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:50252 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:18.693573 7077 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:18.693883 7077 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:18.702735 7077 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:44:18.702853 7077 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:18.702952 7077 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:44:18.703061 7077 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:44:18.716692 7077 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:50252 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:18.724251 7077 master_main.cc:80] Initializing master server... I0517 06:44:18.725739 7077 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 128319us I0517 06:44:18.726477 7077 fs_manager.cc:260] Metadata directory not provided I0517 06:44:18.726599 7077 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:18.726796 7077 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:44:18.726902 7077 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:18.734067 7077 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "5cf0d2389b82497caf635141f8800e65" format_stamp: "Formatted at 2018-05-17 06:44:18 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:18.734710 7077 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "5cf0d2389b82497caf635141f8800e65" format_stamp: "Formatted at 2018-05-17 06:44:18 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:18.744123 7077 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:44:18.745268 7077 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:18.745654 7077 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:18.751935 7077 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.004s I0517 06:44:18.752041 7077 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "5cf0d2389b82497caf635141f8800e65" format_stamp: "Formatted at 2018-05-17 06:44:18 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:18.752228 7077 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:18.925081 7077 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:18.926739 7077 master_main.cc:83] Starting Master server... I0517 06:44:18.966586 7077 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:50252 I0517 06:44:18.967414 7077 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:44:18.967512 7077 webserver.cc:184] Document root disabled I0517 06:44:18.968708 7077 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:52951/ I0517 06:44:18.969429 7077 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:44:18.971978 7138 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:18.974226 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 7077 I0517 06:44:18.975535 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50252 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:18.987915 7138 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: Bootstrap starting. I0517 06:44:18.990150 7138 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: No blocks or log segments found. Creating new log. I0517 06:44:18.991276 7138 log.cc:520] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: Log is configured to *not* fsync() on all Append() calls I0517 06:44:18.993587 7138 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: No bootstrap required, opened a new log I0517 06:44:19.005187 7138 raft_consensus.cc:303] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } I0517 06:44:19.005488 7138 raft_consensus.cc:329] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:19.005614 7138 raft_consensus.cc:604] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf0d2389b82497caf635141f8800e65, State: Initialized, Role: FOLLOWER I0517 06:44:19.006072 7138 consensus_queue.cc:227] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } I0517 06:44:19.006441 7138 raft_consensus.cc:343] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:44:19.006582 7138 raft_consensus.cc:435] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:44:19.006733 7138 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: Snoozing failure detection for 1.901s (starting election) I0517 06:44:19.006870 7138 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:19.009248 7138 raft_consensus.cc:457] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } I0517 06:44:19.009649 7138 leader_election.cc:270] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:44:19.011241 7141 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: Snoozing failure detection for 1.610s (election complete) I0517 06:44:19.011523 7141 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:44:19.022383 7141 raft_consensus.cc:571] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 1 LEADER]: Becoming Leader. State: Replica: 5cf0d2389b82497caf635141f8800e65, State: Running, Role: LEADER I0517 06:44:19.022792 7141 consensus_queue.cc:204] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } I0517 06:44:19.026010 7142 sys_catalog.cc:335] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cf0d2389b82497caf635141f8800e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } } I0517 06:44:19.026458 7142 sys_catalog.cc:338] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: This master's current role is: LEADER WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:19.087932 7139 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:19.088279 7139 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:19.096210 7139 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:19.096539 7139 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:44:19.028064 7138 sys_catalog.cc:335] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "5cf0d2389b82497caf635141f8800e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } } I0517 06:44:19.109670 7138 sys_catalog.cc:338] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: This master's current role is: LEADER I0517 06:44:19.110602 7138 sys_catalog.cc:422] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:44:19.029196 7143 sys_catalog.cc:335] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cf0d2389b82497caf635141f8800e65. Latest consensus state: current_term: 1 leader_uuid: "5cf0d2389b82497caf635141f8800e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } } I0517 06:44:19.113701 7143 sys_catalog.cc:338] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: This master's current role is: LEADER I0517 06:44:19.120090 7149 catalog_manager.cc:504] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0517 06:44:19.120656 7146 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:44:19.120785 7077 master_main.cc:86] Master server successfully started. I0517 06:44:19.120862 7139 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50252 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:19.124521 7146 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:44:19.133508 7139 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:19.135943 7139 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 128819us I0517 06:44:19.136562 7139 fs_manager.cc:260] Metadata directory not provided I0517 06:44:19.136731 7139 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:19.137003 7139 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:44:19.137179 7139 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:19.150416 7139 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "489338f6c6514088a81ec10d776c2d45" format_stamp: "Formatted at 2018-05-17 06:44:19 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:19.151378 7139 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "489338f6c6514088a81ec10d776c2d45" format_stamp: "Formatted at 2018-05-17 06:44:19 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:19.167373 7139 fs_manager.cc:495] Time spent creating directory manager: real 0.016s user 0.020s sys 0.000s I0517 06:44:19.168572 7139 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:19.169003 7139 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:19.176344 7146 catalog_manager.cc:899] Generated new certificate authority record I0517 06:44:19.178741 7146 catalog_manager.cc:1021] Loading token signing keys... I0517 06:44:19.191071 7139 fs_manager.cc:417] Time spent opening block manager: real 0.014s user 0.004s sys 0.012s I0517 06:44:19.191200 7139 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "489338f6c6514088a81ec10d776c2d45" format_stamp: "Formatted at 2018-05-17 06:44:19 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:19.191402 7139 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:19.202096 7146 catalog_manager.cc:3962] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: Generated new TSK 0 I0517 06:44:19.419867 7139 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:19.422435 7139 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:19.423545 7139 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:19.525182 7139 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:57754 I0517 06:44:19.526111 7139 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:44:19.526219 7139 webserver.cc:184] Document root disabled I0517 06:44:19.527580 7139 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:45919/ I0517 06:44:19.528273 7139 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:44:19.532991 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 7139 I0517 06:44:19.535197 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50252 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:19.542659 7139 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:19.559995 7271 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50252 I0517 06:44:19.560292 7271 heartbeater.cc:426] Registering TS with master... I0517 06:44:19.560863 7271 heartbeater.cc:475] Master 127.4.96.62:50252 requested a full tablet report, sending... I0517 06:44:19.562865 7105 ts_manager.cc:81] Registered new tserver with Master: 489338f6c6514088a81ec10d776c2d45 (127.4.96.1:57754) I0517 06:44:19.565356 7105 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:39194 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:19.644634 7275 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:19.644981 7275 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:19.652981 7275 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:19.653312 7275 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:44:19.666278 7275 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50252 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:19.674348 7275 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:19.676105 7275 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 128819us I0517 06:44:19.676543 7275 fs_manager.cc:260] Metadata directory not provided I0517 06:44:19.676662 7275 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:19.676863 7275 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:44:19.676982 7275 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:19.684629 7275 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "e79da8f222574a1194bdfa49fb5dda93" format_stamp: "Formatted at 2018-05-17 06:44:19 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:19.685333 7275 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "e79da8f222574a1194bdfa49fb5dda93" format_stamp: "Formatted at 2018-05-17 06:44:19 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:19.695370 7275 fs_manager.cc:495] Time spent creating directory manager: real 0.010s user 0.004s sys 0.012s I0517 06:44:19.696494 7275 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:19.696903 7275 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:19.703450 7275 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0517 06:44:19.703562 7275 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "e79da8f222574a1194bdfa49fb5dda93" format_stamp: "Formatted at 2018-05-17 06:44:19 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:19.703768 7275 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:19.996624 7275 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:19.999194 7275 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:20.000335 7275 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:20.102948 7275 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:48869 I0517 06:44:20.103860 7275 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:44:20.103967 7275 webserver.cc:184] Document root disabled I0517 06:44:20.105319 7275 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:54373/ I0517 06:44:20.106079 7275 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:44:20.108702 7275 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:20.112242 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 7275 I0517 06:44:20.113454 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50252 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:20.132133 7396 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50252 I0517 06:44:20.132465 7396 heartbeater.cc:426] Registering TS with master... I0517 06:44:20.133019 7396 heartbeater.cc:475] Master 127.4.96.62:50252 requested a full tablet report, sending... I0517 06:44:20.134665 7105 ts_manager.cc:81] Registered new tserver with Master: e79da8f222574a1194bdfa49fb5dda93 (127.4.96.2:48869) I0517 06:44:20.136586 7105 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:33895 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:20.223922 7400 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:20.224263 7400 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:20.232197 7400 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:20.232527 7400 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:44:20.245380 7400 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50252 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:20.253604 7400 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:20.255427 7400 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 129319us I0517 06:44:20.255868 7400 fs_manager.cc:260] Metadata directory not provided I0517 06:44:20.256000 7400 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:20.256204 7400 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:44:20.256323 7400 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:20.263909 7400 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "06f454a655e2491bba20fcbae6b6904f" format_stamp: "Formatted at 2018-05-17 06:44:20 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:20.264634 7400 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "06f454a655e2491bba20fcbae6b6904f" format_stamp: "Formatted at 2018-05-17 06:44:20 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:20.274330 7400 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:44:20.275447 7400 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:20.275848 7400 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:20.282433 7400 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0517 06:44:20.282546 7400 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "06f454a655e2491bba20fcbae6b6904f" format_stamp: "Formatted at 2018-05-17 06:44:20 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:20.282740 7400 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:20.500341 7400 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:20.502674 7400 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:20.503594 7400 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:20.568892 7271 heartbeater.cc:467] Master 127.4.96.62:50252 was elected leader, sending a full tablet report... I0517 06:44:20.588752 7400 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:38778 I0517 06:44:20.589601 7400 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:44:20.589699 7400 webserver.cc:184] Document root disabled I0517 06:44:20.590896 7400 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:49828/ I0517 06:44:20.591575 7400 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:44:20.594084 7400 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:20.601056 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 7400 I0517 06:44:20.602350 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50252 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:20.613786 7521 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50252 I0517 06:44:20.614028 7521 heartbeater.cc:426] Registering TS with master... I0517 06:44:20.614477 7521 heartbeater.cc:475] Master 127.4.96.62:50252 requested a full tablet report, sending... I0517 06:44:20.615828 7105 ts_manager.cc:81] Registered new tserver with Master: 06f454a655e2491bba20fcbae6b6904f (127.4.96.3:38778) I0517 06:44:20.616998 7105 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:46066 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:20.706285 7524 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:20.706605 7524 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:20.714470 7524 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:20.714798 7524 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:44:20.727202 7524 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50252 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:20.735235 7524 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:20.736954 7524 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 129319us I0517 06:44:20.737377 7524 fs_manager.cc:260] Metadata directory not provided I0517 06:44:20.737496 7524 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:20.737687 7524 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:44:20.737802 7524 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:20.745198 7524 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "c77c835e5d174d04b58e42e63029fc67" format_stamp: "Formatted at 2018-05-17 06:44:20 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:20.745832 7524 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "c77c835e5d174d04b58e42e63029fc67" format_stamp: "Formatted at 2018-05-17 06:44:20 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:20.755167 7524 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:44:20.756260 7524 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:20.756656 7524 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:20.762936 7524 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.004s I0517 06:44:20.763049 7524 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "c77c835e5d174d04b58e42e63029fc67" format_stamp: "Formatted at 2018-05-17 06:44:20 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:20.763257 7524 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:20.913102 7524 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:20.915428 7524 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:20.916332 7524 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:21.008863 7524 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:53187 I0517 06:44:21.009740 7524 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:44:21.009847 7524 webserver.cc:184] Document root disabled I0517 06:44:21.011041 7524 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:43084/ I0517 06:44:21.011714 7524 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:44:21.014178 7524 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:21.018635 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 7524 I0517 06:44:21.033504 7645 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50252 I0517 06:44:21.033743 7645 heartbeater.cc:426] Registering TS with master... I0517 06:44:21.034184 7645 heartbeater.cc:475] Master 127.4.96.62:50252 requested a full tablet report, sending... I0517 06:44:21.035642 7105 ts_manager.cc:81] Registered new tserver with Master: c77c835e5d174d04b58e42e63029fc67 (127.4.96.4:53187) I0517 06:44:21.036836 7105 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:43955 I0517 06:44:21.038882 4480 external_mini_cluster.cc:482] 4 TS(s) registered with all masters I0517 06:44:21.055265 7105 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41014: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:44:21.093206 7333 tablet_service.cc:759] Processing CreateTablet for tablet dee05aadc1ee46919ccdff3814db7dc6 (table=TestTable [id=338e7ecfdbbd4fa6afeb34a2086e4ad8]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:21.094079 7333 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dee05aadc1ee46919ccdff3814db7dc6. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:21.111821 7333 ts_tablet_manager.cc:1173] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:21.112191 7458 tablet_service.cc:759] Processing CreateTablet for tablet dee05aadc1ee46919ccdff3814db7dc6 (table=TestTable [id=338e7ecfdbbd4fa6afeb34a2086e4ad8]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:21.113054 7458 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dee05aadc1ee46919ccdff3814db7dc6. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:21.114369 7665 ts_tablet_manager.cc:1028] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93: Bootstrapping tablet I0517 06:44:21.114871 7665 tablet_bootstrap.cc:436] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93: Bootstrap starting. I0517 06:44:21.117787 7665 tablet_bootstrap.cc:587] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93: No blocks or log segments found. Creating new log. I0517 06:44:21.117740 7582 tablet_service.cc:759] Processing CreateTablet for tablet dee05aadc1ee46919ccdff3814db7dc6 (table=TestTable [id=338e7ecfdbbd4fa6afeb34a2086e4ad8]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:21.118410 7582 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dee05aadc1ee46919ccdff3814db7dc6. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:21.119426 7665 log.cc:520] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93: Log is configured to *not* fsync() on all Append() calls I0517 06:44:21.122160 7665 tablet_bootstrap.cc:436] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93: No bootstrap required, opened a new log I0517 06:44:21.122530 7665 ts_tablet_manager.cc:1045] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s I0517 06:44:21.129384 7458 ts_tablet_manager.cc:1173] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:21.129992 7582 ts_tablet_manager.cc:1173] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:21.131042 7667 ts_tablet_manager.cc:1028] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Bootstrapping tablet I0517 06:44:21.131489 7667 tablet_bootstrap.cc:436] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Bootstrap starting. I0517 06:44:21.134140 7667 tablet_bootstrap.cc:587] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: No blocks or log segments found. Creating new log. I0517 06:44:21.135557 7665 raft_consensus.cc:303] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48869 } } I0517 06:44:21.136059 7667 log.cc:520] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Log is configured to *not* fsync() on all Append() calls I0517 06:44:21.136215 7665 raft_consensus.cc:329] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:21.136448 7665 raft_consensus.cc:604] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79da8f222574a1194bdfa49fb5dda93, State: Initialized, Role: FOLLOWER I0517 06:44:21.137150 7665 consensus_queue.cc:227] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48869 } } I0517 06:44:21.138761 7668 ts_tablet_manager.cc:1028] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: Bootstrapping tablet I0517 06:44:21.139116 7668 tablet_bootstrap.cc:436] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: Bootstrap starting. I0517 06:44:21.141345 7668 tablet_bootstrap.cc:587] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: No blocks or log segments found. Creating new log. I0517 06:44:21.142522 7668 log.cc:520] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: Log is configured to *not* fsync() on all Append() calls I0517 06:44:21.148247 7668 tablet_bootstrap.cc:436] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: No bootstrap required, opened a new log I0517 06:44:21.148619 7668 ts_tablet_manager.cc:1045] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s I0517 06:44:21.149551 7667 tablet_bootstrap.cc:436] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: No bootstrap required, opened a new log I0517 06:44:21.149947 7667 ts_tablet_manager.cc:1045] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.016s I0517 06:44:21.164674 7667 raft_consensus.cc:303] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48869 } } I0517 06:44:21.165454 7667 raft_consensus.cc:329] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:21.165769 7667 raft_consensus.cc:604] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f454a655e2491bba20fcbae6b6904f, State: Initialized, Role: FOLLOWER I0517 06:44:21.166467 7667 consensus_queue.cc:227] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48869 } } I0517 06:44:21.167268 7668 raft_consensus.cc:303] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48869 } } I0517 06:44:21.168031 7668 raft_consensus.cc:329] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:21.168234 7668 raft_consensus.cc:604] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77c835e5d174d04b58e42e63029fc67, State: Initialized, Role: FOLLOWER I0517 06:44:21.168851 7668 consensus_queue.cc:227] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48869 } } I0517 06:44:21.174335 7521 heartbeater.cc:467] Master 127.4.96.62:50252 was elected leader, sending a full tablet report... I0517 06:44:21.180830 7667 ts_tablet_manager.cc:1073] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Time spent starting tablet: real 0.031s user 0.020s sys 0.004s I0517 06:44:21.182034 7645 heartbeater.cc:467] Master 127.4.96.62:50252 was elected leader, sending a full tablet report... I0517 06:44:21.182584 7668 ts_tablet_manager.cc:1073] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: Time spent starting tablet: real 0.034s user 0.016s sys 0.012s I0517 06:44:21.139498 7396 heartbeater.cc:467] Master 127.4.96.62:50252 was elected leader, sending a full tablet report... I0517 06:44:21.231106 7665 ts_tablet_manager.cc:1073] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93: Time spent starting tablet: real 0.108s user 0.024s sys 0.072s I0517 06:44:21.231510 7670 raft_consensus.cc:435] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:44:21.231796 7670 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93: Snoozing failure detection for 1.718s (starting election) I0517 06:44:21.232076 7670 raft_consensus.cc:457] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48869 } } I0517 06:44:21.233309 7670 leader_election.cc:243] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 06f454a655e2491bba20fcbae6b6904f (127.4.96.3:38778) I0517 06:44:21.233850 7670 leader_election.cc:243] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c77c835e5d174d04b58e42e63029fc67 (127.4.96.4:53187) I0517 06:44:21.246739 7478 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dee05aadc1ee46919ccdff3814db7dc6" candidate_uuid: "e79da8f222574a1194bdfa49fb5dda93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f454a655e2491bba20fcbae6b6904f" is_pre_election: true I0517 06:44:21.247170 7478 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Snoozing failure detection for 2.197s (vote granted) I0517 06:44:21.247367 7478 raft_consensus.cc:2219] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79da8f222574a1194bdfa49fb5dda93 in term 0. I0517 06:44:21.248229 7287 leader_election.cc:379] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 06f454a655e2491bba20fcbae6b6904f (127.4.96.3:38778) I0517 06:44:21.248409 7287 leader_election.cc:270] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:44:21.248874 7674 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93: Snoozing failure detection for 2.098s (election complete) I0517 06:44:21.249079 7674 raft_consensus.cc:2511] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:44:21.249228 7674 raft_consensus.cc:435] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:44:21.249377 7674 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93: Snoozing failure detection for 1.599s (starting election) I0517 06:44:21.249550 7674 raft_consensus.cc:2756] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:21.252337 7602 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dee05aadc1ee46919ccdff3814db7dc6" candidate_uuid: "e79da8f222574a1194bdfa49fb5dda93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77c835e5d174d04b58e42e63029fc67" is_pre_election: true I0517 06:44:21.252741 7602 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: Snoozing failure detection for 2.110s (vote granted) I0517 06:44:21.252913 7602 raft_consensus.cc:2219] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79da8f222574a1194bdfa49fb5dda93 in term 0. I0517 06:44:21.253129 7674 raft_consensus.cc:457] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48869 } } I0517 06:44:21.253664 7287 leader_election.cc:379] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [CANDIDATE]: Term 1 pre-election: Vote granted by peer c77c835e5d174d04b58e42e63029fc67 (127.4.96.4:53187) I0517 06:44:21.253978 7674 leader_election.cc:243] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [CANDIDATE]: Term 1 election: Requesting vote from peer 06f454a655e2491bba20fcbae6b6904f (127.4.96.3:38778) I0517 06:44:21.254335 7674 leader_election.cc:243] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [CANDIDATE]: Term 1 election: Requesting vote from peer c77c835e5d174d04b58e42e63029fc67 (127.4.96.4:53187) I0517 06:44:21.255059 7478 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dee05aadc1ee46919ccdff3814db7dc6" candidate_uuid: "e79da8f222574a1194bdfa49fb5dda93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f454a655e2491bba20fcbae6b6904f" I0517 06:44:21.255259 7602 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dee05aadc1ee46919ccdff3814db7dc6" candidate_uuid: "e79da8f222574a1194bdfa49fb5dda93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77c835e5d174d04b58e42e63029fc67" I0517 06:44:21.255370 7478 raft_consensus.cc:2756] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:21.255563 7478 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Snoozing failure detection for 2.108s (vote granted) I0517 06:44:21.255565 7602 raft_consensus.cc:2756] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:21.255758 7602 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: Snoozing failure detection for 2.215s (vote granted) I0517 06:44:21.258255 7478 raft_consensus.cc:2219] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79da8f222574a1194bdfa49fb5dda93 in term 1. I0517 06:44:21.258424 7602 raft_consensus.cc:2219] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79da8f222574a1194bdfa49fb5dda93 in term 1. I0517 06:44:21.258903 7287 leader_election.cc:379] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [CANDIDATE]: Term 1 election: Vote granted by peer 06f454a655e2491bba20fcbae6b6904f (127.4.96.3:38778) I0517 06:44:21.259055 7287 leader_election.cc:270] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:44:21.259352 7674 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93: Snoozing failure detection for 1.577s (election complete) I0517 06:44:21.259470 7287 leader_election.cc:379] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [CANDIDATE]: Term 1 election: Vote granted by peer c77c835e5d174d04b58e42e63029fc67 (127.4.96.4:53187) I0517 06:44:21.259521 7674 raft_consensus.cc:2511] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:44:21.259775 7674 raft_consensus.cc:571] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [term 1 LEADER]: Becoming Leader. State: Replica: e79da8f222574a1194bdfa49fb5dda93, State: Running, Role: LEADER I0517 06:44:21.260087 7674 consensus_queue.cc:204] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48869 } } I0517 06:44:21.265028 7105 catalog_manager.cc:3686] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 reported cstate change: term changed from 0 to 1, leader changed from to e79da8f222574a1194bdfa49fb5dda93 (127.4.96.2). New cstate: current_term: 1 leader_uuid: "e79da8f222574a1194bdfa49fb5dda93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48869 } health_report { overall_health: HEALTHY } } } I0517 06:44:21.284201 4480 external_mini_cluster.cc:482] 4 TS(s) registered with all masters I0517 06:44:21.286553 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 489338f6c6514088a81ec10d776c2d45 to finish bootstrapping I0517 06:44:21.302480 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e79da8f222574a1194bdfa49fb5dda93 to finish bootstrapping I0517 06:44:21.312018 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 06f454a655e2491bba20fcbae6b6904f to finish bootstrapping I0517 06:44:21.322240 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c77c835e5d174d04b58e42e63029fc67 to finish bootstrapping I0517 06:44:21.331552 4480 kudu-admin-test.cc:561] Waiting for Master to see the current replicas... I0517 06:44:21.333710 4480 kudu-admin-test.cc:568] Tablet locations: tablet_id: "dee05aadc1ee46919ccdff3814db7dc6" replicas { ts_info { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" rpc_addresses { host: "127.4.96.3" port: 38778 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" rpc_addresses { host: "127.4.96.4" port: 53187 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" rpc_addresses { host: "127.4.96.2" port: 48869 } } role: LEADER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } W0517 06:44:21.345811 7522 tablet.cc:1678] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:44:21.265604 7646 tablet.cc:1678] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:44:21.360626 7397 tablet.cc:1678] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:21.734182 7602 raft_consensus.cc:1057] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 1 FOLLOWER]: Refusing update from remote peer e79da8f222574a1194bdfa49fb5dda93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:21.735296 7674 consensus_queue.cc:963] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:21.742841 7478 raft_consensus.cc:1057] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 1 FOLLOWER]: Refusing update from remote peer e79da8f222574a1194bdfa49fb5dda93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:21.744500 7674 consensus_queue.cc:963] T dee05aadc1ee46919ccdff3814db7dc6 P e79da8f222574a1194bdfa49fb5dda93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:21.753365 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 7275 W0517 06:44:21.771076 7086 connection.cc:512] client connection to 127.4.96.2:48869 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:48869): Connection reset by peer (error 104) W0517 06:44:21.771134 7076 connection.cc:512] client connection to 127.4.96.2:48869 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:48869): Connection reset by peer (error 104) W0517 06:44:21.771185 7408 connection.cc:512] server connection from 127.4.96.2:43239 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:43239): Connection reset by peer (error 104) W0517 06:44:21.772020 7086 connection.cc:512] server connection from 127.4.96.2:33895 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:33895): Connection reset by peer (error 104) W0517 06:44:21.772476 7073 connection.cc:512] client connection to 127.4.96.2:48869 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:48869): Connection reset by peer (error 104) W0517 06:44:21.772531 7533 connection.cc:512] server connection from 127.4.96.2:58953 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:58953): Connection reset by peer (error 104) I0517 06:44:21.773439 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 7077 W0517 06:44:21.796303 7412 connection.cc:512] client connection to 127.4.96.62:50252 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50252): Connection reset by peer (error 104) W0517 06:44:21.796366 7162 connection.cc:512] client connection to 127.4.96.62:50252 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50252): Connection reset by peer (error 104) I0517 06:44:21.796941 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:50252 --webserver_interface=127.4.96.62 --webserver_port=52951 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} W0517 06:44:21.797488 7536 connection.cc:512] server connection from 127.0.0.1:43606 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43606): Connection reset by peer (error 104) W0517 06:44:21.798192 7536 connection.cc:512] client connection to 127.4.96.62:50252 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50252): Connection reset by peer (error 104) W0517 06:44:21.799039 7408 connection.cc:512] server connection from 127.0.0.1:54902 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:54902): Connection reset by peer (error 104) W0517 06:44:21.802064 7076 connection.cc:512] client connection to 127.4.96.62:50252 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50252): Connection reset by peer (error 104) W0517 06:44:21.806162 7653 connection.cc:512] client connection to 127.4.96.62:50252 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50252): Connection reset by peer (error 104) WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:21.910537 7684 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:21.910845 7684 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:21.919823 7684 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:44:21.919940 7684 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:21.920037 7684 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:44:21.920142 7684 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:44:21.933895 7684 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:50252 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=52951 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:21.941529 7684 master_main.cc:80] Initializing master server... I0517 06:44:21.943037 7684 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 129819us I0517 06:44:21.943769 7684 fs_manager.cc:260] Metadata directory not provided I0517 06:44:21.943903 7684 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:21.949249 7684 fs_manager.cc:397] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s I0517 06:44:21.949854 7684 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:21.950259 7684 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:21.956681 7684 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0517 06:44:21.956790 7684 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "5cf0d2389b82497caf635141f8800e65" format_stamp: "Formatted at 2018-05-17 06:44:18 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:21.957939 7684 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:22.125134 7684 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:22.126894 7684 master_main.cc:83] Starting Master server... I0517 06:44:22.166466 7684 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:50252 I0517 06:44:22.167309 7684 webserver.cc:173] Starting webserver on 127.4.96.62:52951 I0517 06:44:22.167407 7684 webserver.cc:184] Document root disabled I0517 06:44:22.168570 7684 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:52951/ I0517 06:44:22.171486 7684 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:44:22.180464 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 7684 I0517 06:44:22.180549 7745 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: Bootstrap starting. I0517 06:44:22.180778 4480 kudu-admin-test.cc:585] Forcing unsafe config change on tserver 06f454a655e2491bba20fcbae6b6904f I0517 06:44:22.204787 7745 log.cc:520] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: Log is configured to *not* fsync() on all Append() calls I0517 06:44:22.216362 7745 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:22.216920 7745 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: Bootstrap complete. I0517 06:44:22.235518 7745 raft_consensus.cc:303] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } I0517 06:44:22.237080 7745 raft_consensus.cc:604] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf0d2389b82497caf635141f8800e65, State: Initialized, Role: FOLLOWER I0517 06:44:22.237584 7745 consensus_queue.cc:227] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } I0517 06:44:22.237999 7745 raft_consensus.cc:343] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:44:22.238175 7745 raft_consensus.cc:435] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:44:22.238360 7745 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: Snoozing failure detection for 1.744s (starting election) I0517 06:44:22.238548 7745 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:44:22.243010 7745 raft_consensus.cc:457] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } I0517 06:44:22.243420 7745 leader_election.cc:270] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0517 06:44:22.244992 7748 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: Snoozing failure detection for 2.174s (election complete) I0517 06:44:22.245272 7748 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 2 FOLLOWER]: Leader election won for term 2 I0517 06:44:22.254427 7748 raft_consensus.cc:571] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [term 2 LEADER]: Becoming Leader. State: Replica: 5cf0d2389b82497caf635141f8800e65, State: Running, Role: LEADER I0517 06:44:22.254834 7748 consensus_queue.cc:204] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } I0517 06:44:22.258054 7745 sys_catalog.cc:335] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "5cf0d2389b82497caf635141f8800e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } } I0517 06:44:22.258411 7745 sys_catalog.cc:338] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: This master's current role is: LEADER I0517 06:44:22.260192 7745 sys_catalog.cc:422] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:44:22.262142 7684 master_main.cc:86] Master server successfully started. W0517 06:44:22.302384 7746 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:44:22.261500 7749 sys_catalog.cc:335] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5cf0d2389b82497caf635141f8800e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } } I0517 06:44:22.321616 7749 sys_catalog.cc:338] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: This master's current role is: LEADER I0517 06:44:22.261833 7750 sys_catalog.cc:335] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cf0d2389b82497caf635141f8800e65. Latest consensus state: current_term: 2 leader_uuid: "5cf0d2389b82497caf635141f8800e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf0d2389b82497caf635141f8800e65" member_type: VOTER } } I0517 06:44:22.323835 7750 sys_catalog.cc:338] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65 [sys.catalog]: This master's current role is: LEADER I0517 06:44:22.266116 7752 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:44:22.329938 7752 catalog_manager.cc:321] Loaded metadata for table TestTable [id=338e7ecfdbbd4fa6afeb34a2086e4ad8] I0517 06:44:22.331831 7752 catalog_manager.cc:375] Loaded metadata for tablet dee05aadc1ee46919ccdff3814db7dc6 (table TestTable [id=338e7ecfdbbd4fa6afeb34a2086e4ad8]) I0517 06:44:22.332247 7752 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:44:22.335600 7752 catalog_manager.cc:1021] Loading token signing keys... I0517 06:44:22.337002 7752 catalog_manager.cc:3973] T 00000000000000000000000000000000 P 5cf0d2389b82497caf635141f8800e65: Loaded TSK: 0 I0517 06:44:22.504957 7478 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "06f454a655e2491bba20fcbae6b6904f" tablet_id: "dee05aadc1ee46919ccdff3814db7dc6" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" } peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" } } from {username='slave'} at 127.0.0.1:54911 W0517 06:44:22.505910 7478 raft_consensus.cc:1990] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 1 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48869 } }NEW CONFIG: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } unsafe_config_change: true I0517 06:44:22.506870 7478 raft_consensus.cc:2756] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 1 FOLLOWER]: Advancing to term 2 I0517 06:44:22.537083 7712 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" instance_seqno: 1526539460570725) as {username='slave'} at 127.4.96.3:35753; Asking this server to re-register. I0517 06:44:22.538473 7521 heartbeater.cc:426] Registering TS with master... I0517 06:44:22.538722 7521 heartbeater.cc:475] Master 127.4.96.62:50252 requested a full tablet report, sending... I0517 06:44:22.540369 7712 ts_manager.cc:81] Registered new tserver with Master: 06f454a655e2491bba20fcbae6b6904f (127.4.96.3:38778) I0517 06:44:22.583252 7712 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "489338f6c6514088a81ec10d776c2d45" instance_seqno: 1526539459504354) as {username='slave'} at 127.4.96.1:43530; Asking this server to re-register. I0517 06:44:22.584213 7271 heartbeater.cc:426] Registering TS with master... I0517 06:44:22.584472 7271 heartbeater.cc:475] Master 127.4.96.62:50252 requested a full tablet report, sending... I0517 06:44:22.585575 7712 ts_manager.cc:81] Registered new tserver with Master: 489338f6c6514088a81ec10d776c2d45 (127.4.96.1:57754) I0517 06:44:22.755861 7712 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" instance_seqno: 1526539460990099) as {username='slave'} at 127.4.96.4:53489; Asking this server to re-register. I0517 06:44:22.756762 7645 heartbeater.cc:426] Registering TS with master... I0517 06:44:22.757024 7645 heartbeater.cc:475] Master 127.4.96.62:50252 requested a full tablet report, sending... I0517 06:44:22.758618 7712 ts_manager.cc:81] Registered new tserver with Master: c77c835e5d174d04b58e42e63029fc67 (127.4.96.4:53187) I0517 06:44:23.245468 7776 raft_consensus.cc:435] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e79da8f222574a1194bdfa49fb5dda93) I0517 06:44:23.245710 7776 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: Snoozing failure detection for 1.901s (starting election) I0517 06:44:23.245944 7776 raft_consensus.cc:457] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48869 } } I0517 06:44:23.246878 7776 leader_election.cc:243] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 06f454a655e2491bba20fcbae6b6904f (127.4.96.3:38778) I0517 06:44:23.247299 7776 leader_election.cc:243] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e79da8f222574a1194bdfa49fb5dda93 (127.4.96.2:48869) W0517 06:44:23.251209 7533 leader_election.cc:299] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e79da8f222574a1194bdfa49fb5dda93 (127.4.96.2:48869): Network error: Client connection negotiation failed: client connection to 127.4.96.2:48869: connect: Connection refused (error 111) I0517 06:44:23.259312 7478 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dee05aadc1ee46919ccdff3814db7dc6" candidate_uuid: "c77c835e5d174d04b58e42e63029fc67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "06f454a655e2491bba20fcbae6b6904f" is_pre_election: true I0517 06:44:23.259748 7478 raft_consensus.cc:2180] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c77c835e5d174d04b58e42e63029fc67 for term 2 because replica is either leader or believes a valid leader to be alive. I0517 06:44:23.260573 7536 leader_election.cc:393] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 06f454a655e2491bba20fcbae6b6904f (127.4.96.3:38778). Message: Invalid argument: T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c77c835e5d174d04b58e42e63029fc67 for term 2 because replica is either leader or believes a valid leader to be alive. I0517 06:44:23.260781 7536 leader_election.cc:270] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0517 06:44:23.261176 7776 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: Snoozing failure detection for 1.604s (election complete) I0517 06:44:23.261399 7776 raft_consensus.cc:2756] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:44:23.264133 7776 raft_consensus.cc:2459] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0517 06:44:24.013480 7781 raft_consensus.cc:435] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0517 06:44:24.013716 7781 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Snoozing failure detection for 2.162s (starting election) I0517 06:44:24.013896 7781 raft_consensus.cc:457] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } unsafe_config_change: true I0517 06:44:24.014645 7781 leader_election.cc:243] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer c77c835e5d174d04b58e42e63029fc67 (127.4.96.4:53187) I0517 06:44:24.023470 7602 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dee05aadc1ee46919ccdff3814db7dc6" candidate_uuid: "06f454a655e2491bba20fcbae6b6904f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c77c835e5d174d04b58e42e63029fc67" is_pre_election: true I0517 06:44:24.023828 7602 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: Snoozing failure detection for 2.248s (vote granted) I0517 06:44:24.023983 7602 raft_consensus.cc:2219] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f454a655e2491bba20fcbae6b6904f in term 2. I0517 06:44:24.024665 7412 leader_election.cc:379] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [CANDIDATE]: Term 3 pre-election: Vote granted by peer c77c835e5d174d04b58e42e63029fc67 (127.4.96.4:53187) I0517 06:44:24.024833 7412 leader_election.cc:270] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0517 06:44:24.025197 7781 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Snoozing failure detection for 1.628s (election complete) I0517 06:44:24.025364 7781 raft_consensus.cc:2511] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 2 FOLLOWER]: Leader pre-election won for term 3 I0517 06:44:24.025501 7781 raft_consensus.cc:435] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0517 06:44:24.025642 7781 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Snoozing failure detection for 1.571s (starting election) I0517 06:44:24.025769 7781 raft_consensus.cc:2756] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 2 FOLLOWER]: Advancing to term 3 I0517 06:44:24.028545 7781 raft_consensus.cc:457] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } unsafe_config_change: true I0517 06:44:24.029060 7781 leader_election.cc:243] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [CANDIDATE]: Term 3 election: Requesting vote from peer c77c835e5d174d04b58e42e63029fc67 (127.4.96.4:53187) I0517 06:44:24.029999 7602 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dee05aadc1ee46919ccdff3814db7dc6" candidate_uuid: "06f454a655e2491bba20fcbae6b6904f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c77c835e5d174d04b58e42e63029fc67" I0517 06:44:24.030297 7602 raft_consensus.cc:2756] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 2 FOLLOWER]: Advancing to term 3 I0517 06:44:24.030458 7602 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67: Snoozing failure detection for 2.947s (vote granted) I0517 06:44:24.032948 7602 raft_consensus.cc:2219] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f454a655e2491bba20fcbae6b6904f in term 3. I0517 06:44:24.033552 7412 leader_election.cc:379] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [CANDIDATE]: Term 3 election: Vote granted by peer c77c835e5d174d04b58e42e63029fc67 (127.4.96.4:53187) I0517 06:44:24.033707 7412 leader_election.cc:270] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0517 06:44:24.034009 7781 raft_consensus.cc:2704] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Snoozing failure detection for 1.655s (election complete) I0517 06:44:24.034171 7781 raft_consensus.cc:2511] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 3 FOLLOWER]: Leader election won for term 3 I0517 06:44:24.035023 7781 raft_consensus.cc:571] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 3 LEADER]: Becoming Leader. State: Replica: 06f454a655e2491bba20fcbae6b6904f, State: Running, Role: LEADER I0517 06:44:24.035387 7781 consensus_queue.cc:204] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } unsafe_config_change: true I0517 06:44:24.039409 7712 catalog_manager.cc:3686] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f reported cstate change: term changed from 1 to 3, leader changed from e79da8f222574a1194bdfa49fb5dda93 (127.4.96.2) to 06f454a655e2491bba20fcbae6b6904f (127.4.96.3), now has a pending config: VOTER 06f454a655e2491bba20fcbae6b6904f (127.4.96.3), VOTER c77c835e5d174d04b58e42e63029fc67 (127.4.96.4). New cstate: current_term: 3 leader_uuid: "06f454a655e2491bba20fcbae6b6904f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79da8f222574a1194bdfa49fb5dda93" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48869 } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } unsafe_config_change: true } I0517 06:44:24.623034 7602 raft_consensus.cc:1057] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 3 FOLLOWER]: Refusing update from remote peer 06f454a655e2491bba20fcbae6b6904f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch) I0517 06:44:24.624055 7791 consensus_queue.cc:963] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s I0517 06:44:24.630906 7794 raft_consensus.cc:2658] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 3 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER e79da8f222574a1194bdfa49fb5dda93 (127.4.96.2) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } unsafe_config_change: true } I0517 06:44:24.631958 7602 raft_consensus.cc:2658] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 3 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER e79da8f222574a1194bdfa49fb5dda93 (127.4.96.2) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } unsafe_config_change: true } I0517 06:44:24.636613 7712 catalog_manager.cc:3686] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f reported cstate change: config changed from index -1 to 2, VOTER e79da8f222574a1194bdfa49fb5dda93 (127.4.96.2) evicted, no longer has a pending config: VOTER 06f454a655e2491bba20fcbae6b6904f (127.4.96.3), VOTER c77c835e5d174d04b58e42e63029fc67 (127.4.96.4). New cstate: current_term: 3 leader_uuid: "06f454a655e2491bba20fcbae6b6904f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:44:24.643177 7712 catalog_manager.cc:2791] Scheduling retry of dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93 with a delay of 26 ms (attempt = 1) W0517 06:44:24.643431 7712 catalog_manager.cc:3845] Failed to send dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93: Not found: failed to reset TS proxy: Could not find TS for UUID e79da8f222574a1194bdfa49fb5dda93 I0517 06:44:24.643918 7712 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet dee05aadc1ee46919ccdff3814db7dc6 (attempt 1) I0517 06:44:24.648617 7711 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet dee05aadc1ee46919ccdff3814db7dc6 (attempt 1) I0517 06:44:24.654625 7478 consensus_peers.cc:456] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f -> Peer c77c835e5d174d04b58e42e63029fc67 (127.4.96.4:53187): Closing peer: c77c835e5d174d04b58e42e63029fc67 I0517 06:44:24.654995 7478 consensus_queue.cc:204] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "489338f6c6514088a81ec10d776c2d45" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 57754 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:24.657898 7602 raft_consensus.cc:1057] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 3 FOLLOWER]: Refusing update from remote peer 06f454a655e2491bba20fcbae6b6904f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch) I0517 06:44:24.658723 7696 catalog_manager.cc:3257] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dee05aadc1ee46919ccdff3814db7dc6 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 06f454a655e2491bba20fcbae6b6904f (127.4.96.3:38778) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } unsafe_config_change: true. Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "489338f6c6514088a81ec10d776c2d45" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 57754 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:24.659145 7696 catalog_manager.cc:2791] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dee05aadc1ee46919ccdff3814db7dc6 with cas_config_opid_index 2 with a delay of 21 ms (attempt = 1) I0517 06:44:24.659102 7791 consensus_queue.cc:963] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0517 06:44:24.662799 7794 raft_consensus.cc:2658] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER 489338f6c6514088a81ec10d776c2d45 (127.4.96.1) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "489338f6c6514088a81ec10d776c2d45" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 57754 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:24.664042 7602 raft_consensus.cc:2658] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER 489338f6c6514088a81ec10d776c2d45 (127.4.96.1) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "489338f6c6514088a81ec10d776c2d45" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 57754 } attrs { promote: true } } unsafe_config_change: true } W0517 06:44:24.668848 7408 consensus_peers.cc:434] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f -> Peer 489338f6c6514088a81ec10d776c2d45 (127.4.96.1:57754): Couldn't send request to peer 489338f6c6514088a81ec10d776c2d45 for tablet dee05aadc1ee46919ccdff3814db7dc6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dee05aadc1ee46919ccdff3814db7dc6. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:24.669620 7696 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dee05aadc1ee46919ccdff3814db7dc6 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0517 06:44:24.670099 7711 catalog_manager.cc:3686] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f reported cstate change: config changed from index 2 to 4, NON_VOTER 489338f6c6514088a81ec10d776c2d45 (127.4.96.1) added. New cstate: current_term: 3 leader_uuid: "06f454a655e2491bba20fcbae6b6904f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "489338f6c6514088a81ec10d776c2d45" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 57754 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0517 06:44:24.670789 7692 catalog_manager.cc:2791] Scheduling retry of dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93 with a delay of 52 ms (attempt = 2) W0517 06:44:24.670994 7692 catalog_manager.cc:2810] Async tablet task dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93 failed: Not found: failed to reset TS proxy: Could not find TS for UUID e79da8f222574a1194bdfa49fb5dda93 I0517 06:44:24.681038 7696 catalog_manager.cc:3273] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dee05aadc1ee46919ccdff3814db7dc6 with cas_config_opid_index 2: aborting the task: latest config opid_index 4; task opid_index 2 I0517 06:44:24.723814 7692 catalog_manager.cc:2791] Scheduling retry of dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93 with a delay of 82 ms (attempt = 3) W0517 06:44:24.723984 7692 catalog_manager.cc:2810] Async tablet task dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93 failed: Not found: failed to reset TS proxy: Could not find TS for UUID e79da8f222574a1194bdfa49fb5dda93 I0517 06:44:24.806632 7692 catalog_manager.cc:2791] Scheduling retry of dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93 with a delay of 139 ms (attempt = 4) W0517 06:44:24.806780 7692 catalog_manager.cc:2810] Async tablet task dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93 failed: Not found: failed to reset TS proxy: Could not find TS for UUID e79da8f222574a1194bdfa49fb5dda93 I0517 06:44:24.946409 7692 catalog_manager.cc:2791] Scheduling retry of dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93 with a delay of 298 ms (attempt = 5) W0517 06:44:24.946571 7692 catalog_manager.cc:2810] Async tablet task dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93 failed: Not found: failed to reset TS proxy: Could not find TS for UUID e79da8f222574a1194bdfa49fb5dda93 I0517 06:44:25.127359 7803 ts_tablet_manager.cc:669] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: Initiating tablet copy from peer 06f454a655e2491bba20fcbae6b6904f (127.4.96.3:38778) I0517 06:44:25.127746 7803 tablet_copy_client.cc:239] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.3:38778 I0517 06:44:25.138835 7498 tablet_copy_service.cc:133] P 06f454a655e2491bba20fcbae6b6904f: Received BeginTabletCopySession request for tablet dee05aadc1ee46919ccdff3814db7dc6 from peer 489338f6c6514088a81ec10d776c2d45 ({username='slave'} at 127.4.96.1:33271) I0517 06:44:25.139103 7498 tablet_copy_service.cc:154] P 06f454a655e2491bba20fcbae6b6904f: Beginning new tablet copy session on tablet dee05aadc1ee46919ccdff3814db7dc6 from peer 489338f6c6514088a81ec10d776c2d45 at {username='slave'} at 127.4.96.1:33271: session id = 489338f6c6514088a81ec10d776c2d45-dee05aadc1ee46919ccdff3814db7dc6 I0517 06:44:25.142530 7498 tablet_copy_source_session.cc:202] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:44:25.145503 7803 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dee05aadc1ee46919ccdff3814db7dc6. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:25.156780 7803 ts_tablet_manager.cc:1173] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:44:25.157106 7803 tablet_copy_client.cc:565] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: tablet copy: Starting download of 0 data blocks... I0517 06:44:25.157306 7803 tablet_copy_client.cc:528] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: tablet copy: Starting download of 1 WAL segments... I0517 06:44:25.159215 7803 tablet_copy_client.cc:404] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:44:25.161862 7803 ts_tablet_manager.cc:1028] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: Bootstrapping tablet I0517 06:44:25.162231 7803 tablet_bootstrap.cc:436] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: Bootstrap starting. I0517 06:44:25.167134 7803 log.cc:520] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: Log is configured to *not* fsync() on all Append() calls I0517 06:44:25.172304 7803 tablet_bootstrap.cc:436] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:25.172775 7803 tablet_bootstrap.cc:436] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: Bootstrap complete. I0517 06:44:25.173079 7803 ts_tablet_manager.cc:1045] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s I0517 06:44:25.182607 7803 raft_consensus.cc:303] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "489338f6c6514088a81ec10d776c2d45" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 57754 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:25.183135 7803 raft_consensus.cc:604] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 489338f6c6514088a81ec10d776c2d45, State: Initialized, Role: LEARNER I0517 06:44:25.183413 7803 consensus_queue.cc:227] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "489338f6c6514088a81ec10d776c2d45" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 57754 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:25.185205 7803 ts_tablet_manager.cc:1073] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: Time spent starting tablet: real 0.012s user 0.012s sys 0.004s I0517 06:44:25.186244 7498 tablet_copy_service.cc:323] P 06f454a655e2491bba20fcbae6b6904f: Request end of tablet copy session 489338f6c6514088a81ec10d776c2d45-dee05aadc1ee46919ccdff3814db7dc6 received from {username='slave'} at 127.4.96.1:33271 I0517 06:44:25.186462 7498 tablet_copy_service.cc:411] P 06f454a655e2491bba20fcbae6b6904f: Ending tablet copy session 489338f6c6514088a81ec10d776c2d45-dee05aadc1ee46919ccdff3814db7dc6 on tablet dee05aadc1ee46919ccdff3814db7dc6 with peer 489338f6c6514088a81ec10d776c2d45 I0517 06:44:25.245755 7692 catalog_manager.cc:2791] Scheduling retry of dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93 with a delay of 544 ms (attempt = 6) W0517 06:44:25.245941 7692 catalog_manager.cc:2810] Async tablet task dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93 failed: Not found: failed to reset TS proxy: Could not find TS for UUID e79da8f222574a1194bdfa49fb5dda93 W0517 06:44:25.294980 7272 tablet.cc:1678] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:25.726677 7228 raft_consensus.cc:1002] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[] I0517 06:44:25.790844 7692 catalog_manager.cc:2791] Scheduling retry of dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93 with a delay of 1070 ms (attempt = 7) W0517 06:44:25.791028 7692 catalog_manager.cc:2810] Async tablet task dee05aadc1ee46919ccdff3814db7dc6 Delete Tablet RPC for TS=e79da8f222574a1194bdfa49fb5dda93 failed: Not found: failed to reset TS proxy: Could not find TS for UUID e79da8f222574a1194bdfa49fb5dda93 I0517 06:44:26.218871 7808 raft_consensus.cc:866] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f: attempting to promote NON_VOTER 489338f6c6514088a81ec10d776c2d45 to VOTER I0517 06:44:26.219489 7808 consensus_peers.cc:456] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f -> Peer 489338f6c6514088a81ec10d776c2d45 (127.4.96.1:57754): Closing peer: 489338f6c6514088a81ec10d776c2d45 I0517 06:44:26.219627 7808 consensus_peers.cc:456] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f -> Peer c77c835e5d174d04b58e42e63029fc67 (127.4.96.4:53187): Closing peer: c77c835e5d174d04b58e42e63029fc67 I0517 06:44:26.220005 7808 consensus_queue.cc:204] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "489338f6c6514088a81ec10d776c2d45" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 57754 } attrs { promote: false } } unsafe_config_change: true I0517 06:44:26.223393 7602 raft_consensus.cc:1057] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 3 FOLLOWER]: Refusing update from remote peer 06f454a655e2491bba20fcbae6b6904f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0517 06:44:26.224020 7228 raft_consensus.cc:1057] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45 [term 3 LEARNER]: Refusing update from remote peer 06f454a655e2491bba20fcbae6b6904f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0517 06:44:26.224412 7794 consensus_queue.cc:963] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0517 06:44:26.225013 7807 consensus_queue.cc:963] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [LEADER]: Connected to new peer: Peer: permanent_uuid: "489338f6c6514088a81ec10d776c2d45" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 57754 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0517 06:44:26.236317 7794 raft_consensus.cc:2658] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 489338f6c6514088a81ec10d776c2d45 (127.4.96.1) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "489338f6c6514088a81ec10d776c2d45" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 57754 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:26.237269 7228 raft_consensus.cc:2658] T dee05aadc1ee46919ccdff3814db7dc6 P 489338f6c6514088a81ec10d776c2d45 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 489338f6c6514088a81ec10d776c2d45 (127.4.96.1) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "489338f6c6514088a81ec10d776c2d45" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 57754 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:26.238545 7601 raft_consensus.cc:2658] T dee05aadc1ee46919ccdff3814db7dc6 P c77c835e5d174d04b58e42e63029fc67 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 489338f6c6514088a81ec10d776c2d45 (127.4.96.1) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } } peers { permanent_uuid: "489338f6c6514088a81ec10d776c2d45" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 57754 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:26.244551 7712 catalog_manager.cc:3686] T dee05aadc1ee46919ccdff3814db7dc6 P 06f454a655e2491bba20fcbae6b6904f reported cstate change: config changed from index 4 to 5, 489338f6c6514088a81ec10d776c2d45 (127.4.96.1) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "06f454a655e2491bba20fcbae6b6904f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 38778 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 53187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "489338f6c6514088a81ec10d776c2d45" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 57754 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:44:26.272094 4480 kudu-admin-test.cc:611] Waiting for Master to see new config... I0517 06:44:26.283689 4480 kudu-admin-test.cc:616] Tablet locations: tablet_id: "dee05aadc1ee46919ccdff3814db7dc6" replicas { ts_info { permanent_uuid: "06f454a655e2491bba20fcbae6b6904f" rpc_addresses { host: "127.4.96.3" port: 38778 } } role: LEADER } replicas { ts_info { permanent_uuid: "c77c835e5d174d04b58e42e63029fc67" rpc_addresses { host: "127.4.96.4" port: 53187 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "489338f6c6514088a81ec10d776c2d45" rpc_addresses { host: "127.4.96.1" port: 57754 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0517 06:44:26.284948 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 7139 W0517 06:44:26.306823 7692 connection.cc:512] server connection from 127.4.96.1:43530 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:43530): Connection reset by peer (error 104) W0517 06:44:26.306854 7076 connection.cc:512] client connection to 127.4.96.1:57754 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:57754): Connection reset by peer (error 104) W0517 06:44:26.306915 7072 connection.cc:512] client connection to 127.4.96.1:57754 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:57754): Connection reset by peer (error 104) W0517 06:44:26.307973 7408 connection.cc:512] client connection to 127.4.96.1:57754 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:57754): Connection reset by peer (error 104) W0517 06:44:26.308712 7409 connection.cc:512] server connection from 127.4.96.1:33271 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:33271): Connection reset by peer (error 104) I0517 06:44:26.308846 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 7400 W0517 06:44:26.336699 7696 connection.cc:512] client connection to 127.4.96.3:38778 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:38778): Connection reset by peer (error 104) W0517 06:44:26.336741 7536 connection.cc:512] client connection to 127.4.96.3:38778 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:38778): Connection reset by peer (error 104) W0517 06:44:26.336863 7076 connection.cc:512] client connection to 127.4.96.3:38778 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:38778): Connection reset by peer (error 104) W0517 06:44:26.337450 7692 connection.cc:512] server connection from 127.4.96.3:35753 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:35753): Connection reset by peer (error 104) W0517 06:44:26.337800 7533 connection.cc:512] server connection from 127.4.96.3:38240 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:38240): Connection reset by peer (error 104) I0517 06:44:26.338167 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 7524 W0517 06:44:26.359079 7696 connection.cc:512] server connection from 127.4.96.4:53489 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:53489): Connection reset by peer (error 104) W0517 06:44:26.359100 7076 connection.cc:512] client connection to 127.4.96.4:53187 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:53187): Connection reset by peer (error 104) I0517 06:44:26.359248 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 7684 W0517 06:44:26.386543 7076 connection.cc:512] client connection to 127.4.96.62:50252 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50252): Connection reset by peer (error 104) [ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (7835 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig I0517 06:44:26.411200 4480 test_util.cc:200] Using random seed: 715230377 I0517 06:44:26.414088 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:44:26.414183 4480 ts_itest-base.cc:114] -------------- I0517 06:44:26.414257 4480 ts_itest-base.cc:115] 8 tablet servers I0517 06:44:26.414328 4480 ts_itest-base.cc:116] 5 replicas per TS I0517 06:44:26.414398 4480 ts_itest-base.cc:117] -------------- I0517 06:44:26.416582 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:54038 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:26.528357 7826 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:26.528673 7826 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:26.537534 7826 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:44:26.537659 7826 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:26.537760 7826 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:44:26.537870 7826 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:44:26.551684 7826 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:54038 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:26.559355 7826 master_main.cc:80] Initializing master server... I0517 06:44:26.560856 7826 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 132319us I0517 06:44:26.561594 7826 fs_manager.cc:260] Metadata directory not provided I0517 06:44:26.561717 7826 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:26.561908 7826 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:44:26.562032 7826 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:26.569248 7826 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "748e9fd9bc4b4d48808361d1ce5c7916" format_stamp: "Formatted at 2018-05-17 06:44:26 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:26.569882 7826 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "748e9fd9bc4b4d48808361d1ce5c7916" format_stamp: "Formatted at 2018-05-17 06:44:26 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:26.579329 7826 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:44:26.580449 7826 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:26.580866 7826 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:26.587249 7826 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:44:26.587358 7826 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "748e9fd9bc4b4d48808361d1ce5c7916" format_stamp: "Formatted at 2018-05-17 06:44:26 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:26.587551 7826 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:26.786356 7826 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:26.788015 7826 master_main.cc:83] Starting Master server... I0517 06:44:26.827891 7826 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:54038 I0517 06:44:26.828729 7826 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:44:26.828829 7826 webserver.cc:184] Document root disabled I0517 06:44:26.830005 7826 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:48617/ I0517 06:44:26.830704 7826 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:44:26.833237 7887 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:26.839028 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 7826 I0517 06:44:26.840330 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:26.849068 7887 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: Bootstrap starting. I0517 06:44:26.851387 7887 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: No blocks or log segments found. Creating new log. I0517 06:44:26.852555 7887 log.cc:520] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: Log is configured to *not* fsync() on all Append() calls I0517 06:44:26.854889 7887 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: No bootstrap required, opened a new log I0517 06:44:26.866559 7887 raft_consensus.cc:303] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } I0517 06:44:26.866856 7887 raft_consensus.cc:329] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:26.866987 7887 raft_consensus.cc:604] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748e9fd9bc4b4d48808361d1ce5c7916, State: Initialized, Role: FOLLOWER I0517 06:44:26.867413 7887 consensus_queue.cc:227] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } I0517 06:44:26.867758 7887 raft_consensus.cc:343] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:44:26.867898 7887 raft_consensus.cc:435] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:44:26.868046 7887 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: Snoozing failure detection for 1.690s (starting election) I0517 06:44:26.868186 7887 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:26.870604 7887 raft_consensus.cc:457] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } I0517 06:44:26.871042 7887 leader_election.cc:270] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:44:26.872632 7890 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: Snoozing failure detection for 1.977s (election complete) I0517 06:44:26.872926 7890 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:44:26.883085 7890 raft_consensus.cc:571] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 1 LEADER]: Becoming Leader. State: Replica: 748e9fd9bc4b4d48808361d1ce5c7916, State: Running, Role: LEADER I0517 06:44:26.883496 7890 consensus_queue.cc:204] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } I0517 06:44:26.886943 7887 sys_catalog.cc:335] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } } I0517 06:44:26.887369 7887 sys_catalog.cc:338] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: This master's current role is: LEADER I0517 06:44:26.887136 7892 sys_catalog.cc:335] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 748e9fd9bc4b4d48808361d1ce5c7916. Latest consensus state: current_term: 1 leader_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } } I0517 06:44:26.948842 7892 sys_catalog.cc:338] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: This master's current role is: LEADER I0517 06:44:26.887892 7891 sys_catalog.cc:335] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } } I0517 06:44:26.951786 7891 sys_catalog.cc:338] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: This master's current role is: LEADER I0517 06:44:26.952534 7887 sys_catalog.cc:422] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:44:26.952854 7894 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:44:26.954241 7826 master_main.cc:86] Master server successfully started. I0517 06:44:26.958587 7894 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:26.981658 7888 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:26.982125 7888 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:26.995671 7888 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:26.996001 7888 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:44:27.001155 7894 catalog_manager.cc:899] Generated new certificate authority record I0517 06:44:27.003412 7894 catalog_manager.cc:1021] Loading token signing keys... I0517 06:44:27.015213 7894 catalog_manager.cc:3962] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: Generated new TSK 0 I0517 06:44:27.017549 7888 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:27.026147 7888 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:27.027961 7888 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 132819us I0517 06:44:27.028399 7888 fs_manager.cc:260] Metadata directory not provided I0517 06:44:27.028522 7888 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:27.028746 7888 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:44:27.028872 7888 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:27.036355 7888 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "5c25814d71724ad89c2b2965035156ab" format_stamp: "Formatted at 2018-05-17 06:44:27 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:27.037026 7888 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "5c25814d71724ad89c2b2965035156ab" format_stamp: "Formatted at 2018-05-17 06:44:27 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:27.067126 7888 fs_manager.cc:495] Time spent creating directory manager: real 0.030s user 0.008s sys 0.024s I0517 06:44:27.068264 7888 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:27.068660 7888 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:27.079069 7888 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.004s sys 0.004s I0517 06:44:27.079195 7888 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "5c25814d71724ad89c2b2965035156ab" format_stamp: "Formatted at 2018-05-17 06:44:27 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:27.079399 7888 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:27.341047 7888 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:27.358989 7888 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:27.370254 7888 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:27.623399 7888 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:43055 I0517 06:44:27.624289 7888 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:44:27.624394 7888 webserver.cc:184] Document root disabled I0517 06:44:27.625783 7888 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:34511/ I0517 06:44:27.626520 7888 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:44:27.630205 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 7888 I0517 06:44:27.631353 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:27.633520 7888 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:27.651721 8019 heartbeater.cc:346] Connected to a master server at 127.4.96.62:54038 I0517 06:44:27.651973 8019 heartbeater.cc:426] Registering TS with master... I0517 06:44:27.652415 8019 heartbeater.cc:475] Master 127.4.96.62:54038 requested a full tablet report, sending... I0517 06:44:27.654009 7854 ts_manager.cc:81] Registered new tserver with Master: 5c25814d71724ad89c2b2965035156ab (127.4.96.1:43055) I0517 06:44:27.655704 7854 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:54315 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:27.786273 8023 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:27.786729 8023 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:27.801038 8023 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:27.801537 8023 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:44:27.822015 8023 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:27.834740 8023 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:27.837110 8023 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 132819us I0517 06:44:27.837713 8023 fs_manager.cc:260] Metadata directory not provided I0517 06:44:27.837885 8023 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:27.838171 8023 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:44:27.838335 8023 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:27.850888 8023 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "580b6d55956e4c18bf9297327263345b" format_stamp: "Formatted at 2018-05-17 06:44:27 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:27.851810 8023 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "580b6d55956e4c18bf9297327263345b" format_stamp: "Formatted at 2018-05-17 06:44:27 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:27.865499 8023 fs_manager.cc:495] Time spent creating directory manager: real 0.013s user 0.012s sys 0.000s I0517 06:44:27.867154 8023 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:27.867688 8023 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:27.878001 8023 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:44:27.878154 8023 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "580b6d55956e4c18bf9297327263345b" format_stamp: "Formatted at 2018-05-17 06:44:27 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:27.878422 8023 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:28.081601 8023 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:28.084194 8023 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:28.085302 8023 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:28.187609 8023 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:33256 I0517 06:44:28.188581 8023 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:44:28.188694 8023 webserver.cc:184] Document root disabled I0517 06:44:28.190182 8023 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:39384/ I0517 06:44:28.190891 8023 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:44:28.198820 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 8023 I0517 06:44:28.200232 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:28.214411 8023 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:28.241988 8144 heartbeater.cc:346] Connected to a master server at 127.4.96.62:54038 I0517 06:44:28.242277 8144 heartbeater.cc:426] Registering TS with master... I0517 06:44:28.242835 8144 heartbeater.cc:475] Master 127.4.96.62:54038 requested a full tablet report, sending... I0517 06:44:28.244467 7854 ts_manager.cc:81] Registered new tserver with Master: 580b6d55956e4c18bf9297327263345b (127.4.96.2:33256) I0517 06:44:28.246152 7854 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:44701 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:28.320900 8148 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:28.321224 8148 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:28.329107 8148 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:28.329430 8148 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:44:28.342288 8148 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:28.350277 8148 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:28.352074 8148 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 133319us I0517 06:44:28.352501 8148 fs_manager.cc:260] Metadata directory not provided I0517 06:44:28.352627 8148 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:28.352830 8148 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:44:28.352949 8148 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:28.360395 8148 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "6a3b5a36994a489eb39f15f8700fc089" format_stamp: "Formatted at 2018-05-17 06:44:28 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:28.361047 8148 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "6a3b5a36994a489eb39f15f8700fc089" format_stamp: "Formatted at 2018-05-17 06:44:28 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:28.370385 8148 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:44:28.371464 8148 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:28.371845 8148 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:28.378159 8148 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.004s I0517 06:44:28.378268 8148 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "6a3b5a36994a489eb39f15f8700fc089" format_stamp: "Formatted at 2018-05-17 06:44:28 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:28.378459 8148 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:28.529079 8148 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:28.531440 8148 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:28.532368 8148 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:28.617228 8148 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:43022 I0517 06:44:28.618134 8148 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:44:28.618233 8148 webserver.cc:184] Document root disabled I0517 06:44:28.619413 8148 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:41160/ I0517 06:44:28.620117 8148 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:44:28.622571 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 8148 I0517 06:44:28.623752 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:28.623975 8148 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:28.640842 8269 heartbeater.cc:346] Connected to a master server at 127.4.96.62:54038 I0517 06:44:28.641062 8269 heartbeater.cc:426] Registering TS with master... I0517 06:44:28.641463 8269 heartbeater.cc:475] Master 127.4.96.62:54038 requested a full tablet report, sending... I0517 06:44:28.642841 7854 ts_manager.cc:81] Registered new tserver with Master: 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3:43022) I0517 06:44:28.644457 7854 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:33222 I0517 06:44:28.658113 8019 heartbeater.cc:467] Master 127.4.96.62:54038 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:28.728245 8272 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:28.728559 8272 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:28.736420 8272 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:28.736729 8272 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:44:28.749300 8272 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:28.757218 8272 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:28.758963 8272 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 133319us I0517 06:44:28.759361 8272 fs_manager.cc:260] Metadata directory not provided I0517 06:44:28.759465 8272 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:28.759649 8272 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:44:28.759757 8272 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:28.767107 8272 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "10206d3db30a45a3a4c5a415d3ed2beb" format_stamp: "Formatted at 2018-05-17 06:44:28 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:28.767755 8272 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "10206d3db30a45a3a4c5a415d3ed2beb" format_stamp: "Formatted at 2018-05-17 06:44:28 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:28.777009 8272 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:44:28.778087 8272 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:28.778450 8272 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:28.784708 8272 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:44:28.784847 8272 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "10206d3db30a45a3a4c5a415d3ed2beb" format_stamp: "Formatted at 2018-05-17 06:44:28 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:28.785116 8272 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:28.947739 8272 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:28.950063 8272 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:28.950968 8272 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:29.034963 8272 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:51856 I0517 06:44:29.035835 8272 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:44:29.035931 8272 webserver.cc:184] Document root disabled I0517 06:44:29.037101 8272 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:53196/ I0517 06:44:29.037772 8272 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:44:29.038672 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 8272 I0517 06:44:29.039847 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.4.96.5:0 --local_ip_for_outbound_sockets=127.4.96.5 --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:29.040424 8272 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:29.062517 8393 heartbeater.cc:346] Connected to a master server at 127.4.96.62:54038 I0517 06:44:29.062736 8393 heartbeater.cc:426] Registering TS with master... I0517 06:44:29.063143 8393 heartbeater.cc:475] Master 127.4.96.62:54038 requested a full tablet report, sending... I0517 06:44:29.064486 7854 ts_manager.cc:81] Registered new tserver with Master: 10206d3db30a45a3a4c5a415d3ed2beb (127.4.96.4:51856) I0517 06:44:29.065570 7854 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:38185 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:29.143966 8395 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:29.144299 8395 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:29.152289 8395 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:29.152616 8395 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.5 I0517 06:44:29.164680 8395 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.5 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:29.172843 8395 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:29.174595 8395 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 133819us I0517 06:44:29.175014 8395 fs_manager.cc:260] Metadata directory not provided I0517 06:44:29.175135 8395 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:29.175333 8395 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0517 06:44:29.175458 8395 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:29.182988 8395 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "dafb8d2b9a584a74b41c05787477c5c2" format_stamp: "Formatted at 2018-05-17 06:44:29 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:29.183753 8395 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "dafb8d2b9a584a74b41c05787477c5c2" format_stamp: "Formatted at 2018-05-17 06:44:29 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:29.192785 8395 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:44:29.193858 8395 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:29.194257 8395 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:29.200613 8395 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:44:29.200716 8395 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal uuid: "dafb8d2b9a584a74b41c05787477c5c2" format_stamp: "Formatted at 2018-05-17 06:44:29 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:29.200932 8395 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:29.249220 8144 heartbeater.cc:467] Master 127.4.96.62:54038 was elected leader, sending a full tablet report... I0517 06:44:29.370533 8395 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:29.372905 8395 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:29.373828 8395 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:29.458065 8395 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.5:54438 I0517 06:44:29.458935 8395 webserver.cc:173] Starting webserver on 127.4.96.5:0 I0517 06:44:29.459036 8395 webserver.cc:184] Document root disabled I0517 06:44:29.460314 8395 webserver.cc:311] Webserver started. Bound to: http://127.4.96.5:37367/ I0517 06:44:29.460983 8395 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb I0517 06:44:29.463390 8395 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:29.464612 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 8395 I0517 06:44:29.465654 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.4.96.6:0 --local_ip_for_outbound_sockets=127.4.96.6 --webserver_interface=127.4.96.6 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:29.481675 8517 heartbeater.cc:346] Connected to a master server at 127.4.96.62:54038 I0517 06:44:29.481920 8517 heartbeater.cc:426] Registering TS with master... I0517 06:44:29.482376 8517 heartbeater.cc:475] Master 127.4.96.62:54038 requested a full tablet report, sending... I0517 06:44:29.483732 7854 ts_manager.cc:81] Registered new tserver with Master: dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438) I0517 06:44:29.484810 7854 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.5:44796 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:29.571007 8520 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:29.571401 8520 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:29.579469 8520 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:29.579776 8520 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.6 I0517 06:44:29.592298 8520 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.6:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.4.96.6 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.6 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:29.600352 8520 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:29.602141 8520 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 133819us I0517 06:44:29.602551 8520 fs_manager.cc:260] Metadata directory not provided I0517 06:44:29.602672 8520 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:29.602871 8520 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data/instance: No such file or directory (error 2) I0517 06:44:29.602989 8520 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:29.610357 8520 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data/instance: uuid: "fc2430d5d05d4863b227c5cefc7ac460" format_stamp: "Formatted at 2018-05-17 06:44:29 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:29.611016 8520 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/wal/instance: uuid: "fc2430d5d05d4863b227c5cefc7ac460" format_stamp: "Formatted at 2018-05-17 06:44:29 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:29.620592 8520 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:44:29.621703 8520 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:29.622092 8520 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:29.628473 8520 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:44:29.628584 8520 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/wal uuid: "fc2430d5d05d4863b227c5cefc7ac460" format_stamp: "Formatted at 2018-05-17 06:44:29 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:29.628794 8520 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:29.646723 8269 heartbeater.cc:467] Master 127.4.96.62:54038 was elected leader, sending a full tablet report... I0517 06:44:29.795760 8520 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:29.798156 8520 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:29.799068 8520 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:29.882097 8520 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.6:47664 I0517 06:44:29.882966 8520 webserver.cc:173] Starting webserver on 127.4.96.6:0 I0517 06:44:29.883069 8520 webserver.cc:184] Document root disabled I0517 06:44:29.884287 8520 webserver.cc:311] Webserver started. Bound to: http://127.4.96.6:54589/ I0517 06:44:29.884958 8520 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data/info.pb I0517 06:44:29.888090 8520 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:29.892254 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 8520 I0517 06:44:29.893203 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.4.96.7:0 --local_ip_for_outbound_sockets=127.4.96.7 --webserver_interface=127.4.96.7 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:29.905154 8641 heartbeater.cc:346] Connected to a master server at 127.4.96.62:54038 I0517 06:44:29.905386 8641 heartbeater.cc:426] Registering TS with master... I0517 06:44:29.905813 8641 heartbeater.cc:475] Master 127.4.96.62:54038 requested a full tablet report, sending... I0517 06:44:29.907193 7854 ts_manager.cc:81] Registered new tserver with Master: fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664) I0517 06:44:29.908290 7854 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.6:42937 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:29.995226 8644 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:29.995537 8644 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:30.003543 8644 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:30.003850 8644 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.7 I0517 06:44:30.016074 8644 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.7:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.4.96.7 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.7 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:30.024890 8644 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:30.026643 8644 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 134319us I0517 06:44:30.027050 8644 fs_manager.cc:260] Metadata directory not provided I0517 06:44:30.027158 8644 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:30.027335 8644 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data/instance: No such file or directory (error 2) I0517 06:44:30.027451 8644 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:30.034837 8644 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data/instance: uuid: "51cb8e32a7ed4c7dbd5b79785556c592" format_stamp: "Formatted at 2018-05-17 06:44:30 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:30.035478 8644 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/wal/instance: uuid: "51cb8e32a7ed4c7dbd5b79785556c592" format_stamp: "Formatted at 2018-05-17 06:44:30 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:30.044586 8644 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.012s sys 0.000s I0517 06:44:30.045663 8644 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:30.046048 8644 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:30.052232 8644 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:44:30.052347 8644 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/wal uuid: "51cb8e32a7ed4c7dbd5b79785556c592" format_stamp: "Formatted at 2018-05-17 06:44:30 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:30.052543 8644 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:30.068265 8393 heartbeater.cc:467] Master 127.4.96.62:54038 was elected leader, sending a full tablet report... I0517 06:44:30.229770 8644 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:30.232679 8644 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:30.233513 8644 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:30.353965 8644 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.7:45618 I0517 06:44:30.354856 8644 webserver.cc:173] Starting webserver on 127.4.96.7:0 I0517 06:44:30.354962 8644 webserver.cc:184] Document root disabled I0517 06:44:30.356060 8644 webserver.cc:311] Webserver started. Bound to: http://127.4.96.7:44139/ I0517 06:44:30.356747 8644 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data/info.pb I0517 06:44:30.357488 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 8644 I0517 06:44:30.358646 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data --block_manager=log --rpc_bind_addresses=127.4.96.8:0 --local_ip_for_outbound_sockets=127.4.96.8 --webserver_interface=127.4.96.8 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:30.363656 8644 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:30.382545 8765 heartbeater.cc:346] Connected to a master server at 127.4.96.62:54038 I0517 06:44:30.382774 8765 heartbeater.cc:426] Registering TS with master... I0517 06:44:30.383191 8765 heartbeater.cc:475] Master 127.4.96.62:54038 requested a full tablet report, sending... I0517 06:44:30.384560 7854 ts_manager.cc:81] Registered new tserver with Master: 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7:45618) I0517 06:44:30.386181 7854 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.7:57193 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:30.476621 8768 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:30.476959 8768 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:30.485332 8768 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:30.485651 8768 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.8 I0517 06:44:30.487051 8517 heartbeater.cc:467] Master 127.4.96.62:54038 was elected leader, sending a full tablet report... I0517 06:44:30.500351 8768 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.8:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data/info.pb --webserver_interface=127.4.96.8 --webserver_port=0 --tserver_master_addrs=127.4.96.62:54038 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.8 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:30.508363 8768 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:30.510677 8768 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 134319us I0517 06:44:30.511133 8768 fs_manager.cc:260] Metadata directory not provided I0517 06:44:30.511255 8768 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:30.511502 8768 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data/instance: No such file or directory (error 2) I0517 06:44:30.511615 8768 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:30.519706 8768 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data/instance: uuid: "92255276f9b144f3b14cc51ff4f450de" format_stamp: "Formatted at 2018-05-17 06:44:30 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:30.520325 8768 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/wal/instance: uuid: "92255276f9b144f3b14cc51ff4f450de" format_stamp: "Formatted at 2018-05-17 06:44:30 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:30.531920 8768 fs_manager.cc:495] Time spent creating directory manager: real 0.011s user 0.004s sys 0.008s I0517 06:44:30.533023 8768 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:30.533396 8768 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:30.539659 8768 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:44:30.539772 8768 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/wal uuid: "92255276f9b144f3b14cc51ff4f450de" format_stamp: "Formatted at 2018-05-17 06:44:30 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:30.540830 8768 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:30.733176 8768 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:30.736073 8768 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:30.737545 8768 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:30.871480 8768 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.8:55767 I0517 06:44:30.872373 8768 webserver.cc:173] Starting webserver on 127.4.96.8:0 I0517 06:44:30.872480 8768 webserver.cc:184] Document root disabled I0517 06:44:30.873577 8768 webserver.cc:311] Webserver started. Bound to: http://127.4.96.8:45654/ I0517 06:44:30.874269 8768 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data/info.pb I0517 06:44:30.876958 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 8768 I0517 06:44:30.877182 8768 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:30.910475 8889 heartbeater.cc:346] Connected to a master server at 127.4.96.62:54038 I0517 06:44:30.910634 8641 heartbeater.cc:467] Master 127.4.96.62:54038 was elected leader, sending a full tablet report... I0517 06:44:30.910756 8889 heartbeater.cc:426] Registering TS with master... I0517 06:44:30.911283 8889 heartbeater.cc:475] Master 127.4.96.62:54038 requested a full tablet report, sending... I0517 06:44:30.914458 7853 ts_manager.cc:81] Registered new tserver with Master: 92255276f9b144f3b14cc51ff4f450de (127.4.96.8:55767) I0517 06:44:30.915652 7853 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.8:49119 I0517 06:44:30.919821 4480 external_mini_cluster.cc:482] 8 TS(s) registered with all masters I0517 06:44:30.939110 7853 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57377: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 5 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:44:31.007252 8206 tablet_service.cc:759] Processing CreateTablet for tablet b6bea5c6c58641f8ab69083b20103628 (table=TestTable [id=535082618aca4f96896c631f72424763]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:31.008077 8206 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b6bea5c6c58641f8ab69083b20103628. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:31.024016 8578 tablet_service.cc:759] Processing CreateTablet for tablet b6bea5c6c58641f8ab69083b20103628 (table=TestTable [id=535082618aca4f96896c631f72424763]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:31.024704 8578 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b6bea5c6c58641f8ab69083b20103628. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:31.027673 8206 ts_tablet_manager.cc:1173] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:31.029702 8913 ts_tablet_manager.cc:1028] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089: Bootstrapping tablet I0517 06:44:31.030315 8913 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089: Bootstrap starting. I0517 06:44:31.030223 8454 tablet_service.cc:759] Processing CreateTablet for tablet b6bea5c6c58641f8ab69083b20103628 (table=TestTable [id=535082618aca4f96896c631f72424763]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:31.031179 8454 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b6bea5c6c58641f8ab69083b20103628. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:31.035616 8913 tablet_bootstrap.cc:587] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089: No blocks or log segments found. Creating new log. I0517 06:44:31.037696 8330 tablet_service.cc:759] Processing CreateTablet for tablet b6bea5c6c58641f8ab69083b20103628 (table=TestTable [id=535082618aca4f96896c631f72424763]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:31.038220 8913 log.cc:520] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089: Log is configured to *not* fsync() on all Append() calls I0517 06:44:31.038570 8330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b6bea5c6c58641f8ab69083b20103628. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:31.040331 8702 tablet_service.cc:759] Processing CreateTablet for tablet b6bea5c6c58641f8ab69083b20103628 (table=TestTable [id=535082618aca4f96896c631f72424763]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:31.041424 8702 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b6bea5c6c58641f8ab69083b20103628. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:31.050273 8454 ts_tablet_manager.cc:1173] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:31.059373 8915 ts_tablet_manager.cc:1028] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Bootstrapping tablet I0517 06:44:31.059845 8915 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Bootstrap starting. I0517 06:44:31.060864 8913 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089: No bootstrap required, opened a new log I0517 06:44:31.061086 8330 ts_tablet_manager.cc:1173] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:31.061303 8913 ts_tablet_manager.cc:1045] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.024s I0517 06:44:31.064416 8915 tablet_bootstrap.cc:587] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: No blocks or log segments found. Creating new log. I0517 06:44:31.065878 8915 log.cc:520] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Log is configured to *not* fsync() on all Append() calls I0517 06:44:31.076762 8578 ts_tablet_manager.cc:1173] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:31.078884 8916 ts_tablet_manager.cc:1028] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb: Bootstrapping tablet I0517 06:44:31.079444 8916 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb: Bootstrap starting. I0517 06:44:31.083482 8916 tablet_bootstrap.cc:587] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb: No blocks or log segments found. Creating new log. I0517 06:44:31.085111 8915 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: No bootstrap required, opened a new log I0517 06:44:31.085389 8915 ts_tablet_manager.cc:1045] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.016s I0517 06:44:31.086916 8916 log.cc:520] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb: Log is configured to *not* fsync() on all Append() calls I0517 06:44:31.092486 8913 raft_consensus.cc:303] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.093518 8913 raft_consensus.cc:329] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:31.093448 8702 ts_tablet_manager.cc:1173] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:31.093735 8913 raft_consensus.cc:604] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3b5a36994a489eb39f15f8700fc089, State: Initialized, Role: FOLLOWER I0517 06:44:31.093935 8919 ts_tablet_manager.cc:1028] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Bootstrapping tablet I0517 06:44:31.094442 8919 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Bootstrap starting. I0517 06:44:31.094447 8913 consensus_queue.cc:227] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.097736 8916 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb: No bootstrap required, opened a new log I0517 06:44:31.098130 8916 ts_tablet_manager.cc:1045] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.008s I0517 06:44:31.098942 8920 ts_tablet_manager.cc:1028] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592: Bootstrapping tablet I0517 06:44:31.099426 8920 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592: Bootstrap starting. I0517 06:44:31.099133 8919 tablet_bootstrap.cc:587] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: No blocks or log segments found. Creating new log. I0517 06:44:31.104136 8919 log.cc:520] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Log is configured to *not* fsync() on all Append() calls I0517 06:44:31.103543 8915 raft_consensus.cc:303] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.104486 8920 tablet_bootstrap.cc:587] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592: No blocks or log segments found. Creating new log. I0517 06:44:31.104480 8915 raft_consensus.cc:329] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:31.104666 8915 raft_consensus.cc:604] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dafb8d2b9a584a74b41c05787477c5c2, State: Initialized, Role: FOLLOWER I0517 06:44:31.105307 8915 consensus_queue.cc:227] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.108247 8920 log.cc:520] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592: Log is configured to *not* fsync() on all Append() calls I0517 06:44:31.113178 8919 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: No bootstrap required, opened a new log I0517 06:44:31.113625 8919 ts_tablet_manager.cc:1045] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.008s I0517 06:44:31.115442 8920 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592: No bootstrap required, opened a new log I0517 06:44:31.115797 8920 ts_tablet_manager.cc:1045] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s I0517 06:44:31.118458 8913 ts_tablet_manager.cc:1073] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089: Time spent starting tablet: real 0.057s user 0.020s sys 0.004s I0517 06:44:31.126642 8915 ts_tablet_manager.cc:1073] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Time spent starting tablet: real 0.041s user 0.012s sys 0.020s I0517 06:44:31.129127 8916 raft_consensus.cc:303] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.130089 8916 raft_consensus.cc:329] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:31.130282 8916 raft_consensus.cc:604] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10206d3db30a45a3a4c5a415d3ed2beb, State: Initialized, Role: FOLLOWER I0517 06:44:31.131270 8916 consensus_queue.cc:227] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.134646 8919 raft_consensus.cc:303] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.135486 8919 raft_consensus.cc:329] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:31.135738 8919 raft_consensus.cc:604] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2430d5d05d4863b227c5cefc7ac460, State: Initialized, Role: FOLLOWER I0517 06:44:31.137236 8919 consensus_queue.cc:227] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.138634 8920 raft_consensus.cc:303] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.139472 8920 raft_consensus.cc:329] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:31.139677 8920 raft_consensus.cc:604] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51cb8e32a7ed4c7dbd5b79785556c592, State: Initialized, Role: FOLLOWER I0517 06:44:31.140343 8920 consensus_queue.cc:227] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.146812 8916 ts_tablet_manager.cc:1073] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb: Time spent starting tablet: real 0.048s user 0.012s sys 0.024s I0517 06:44:31.151597 8765 heartbeater.cc:467] Master 127.4.96.62:54038 was elected leader, sending a full tablet report... I0517 06:44:31.153017 8919 ts_tablet_manager.cc:1073] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Time spent starting tablet: real 0.039s user 0.016s sys 0.012s I0517 06:44:31.153913 8920 ts_tablet_manager.cc:1073] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592: Time spent starting tablet: real 0.038s user 0.020s sys 0.012s W0517 06:44:31.126672 8270 tablet.cc:1678] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:31.163450 8925 raft_consensus.cc:435] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:44:31.240696 8925 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb: Snoozing failure detection for 1.907s (starting election) I0517 06:44:31.242233 8925 raft_consensus.cc:457] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.243523 8925 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3:43022) I0517 06:44:31.243988 8925 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438) I0517 06:44:31.251796 8925 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7:45618) I0517 06:44:31.252403 8925 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664) W0517 06:44:31.293272 8394 tablet.cc:1678] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:31.299595 8474 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dafb8d2b9a584a74b41c05787477c5c2" is_pre_election: true I0517 06:44:31.306248 8226 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3b5a36994a489eb39f15f8700fc089" is_pre_election: true I0517 06:44:31.306921 8226 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089: Snoozing failure detection for 1.941s (vote granted) I0517 06:44:31.307201 8226 raft_consensus.cc:2219] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10206d3db30a45a3a4c5a415d3ed2beb in term 0. I0517 06:44:31.308848 8282 leader_election.cc:379] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 pre-election: Vote granted by peer 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3:43022) I0517 06:44:31.314662 8722 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" is_pre_election: true I0517 06:44:31.315340 8722 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592: Snoozing failure detection for 1.618s (vote granted) I0517 06:44:31.315632 8722 raft_consensus.cc:2219] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10206d3db30a45a3a4c5a415d3ed2beb in term 0. I0517 06:44:31.318171 8282 leader_election.cc:379] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 pre-election: Vote granted by peer 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7:45618) I0517 06:44:31.318413 8282 leader_election.cc:270] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:44:31.318938 8925 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb: Snoozing failure detection for 1.683s (election complete) I0517 06:44:31.319924 8925 raft_consensus.cc:2511] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:44:31.320122 8925 raft_consensus.cc:435] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:44:31.320291 8925 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb: Snoozing failure detection for 1.953s (starting election) I0517 06:44:31.320485 8925 raft_consensus.cc:2756] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:31.320648 8598 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc2430d5d05d4863b227c5cefc7ac460" is_pre_election: true W0517 06:44:31.214357 8518 tablet.cc:1678] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:31.326149 8925 raft_consensus.cc:457] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.327651 8925 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 election: Requesting vote from peer 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3:43022) I0517 06:44:31.328682 8925 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 election: Requesting vote from peer dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438) I0517 06:44:31.329648 8226 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3b5a36994a489eb39f15f8700fc089" I0517 06:44:31.330065 8226 raft_consensus.cc:2756] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:31.330293 8226 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089: Snoozing failure detection for 1.535s (vote granted) I0517 06:44:31.335940 8226 raft_consensus.cc:2219] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10206d3db30a45a3a4c5a415d3ed2beb in term 1. I0517 06:44:31.338544 8473 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dafb8d2b9a584a74b41c05787477c5c2" I0517 06:44:31.337054 8282 leader_election.cc:379] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 election: Vote granted by peer 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3:43022) I0517 06:44:31.340682 8925 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 election: Requesting vote from peer 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7:45618) I0517 06:44:31.341135 8925 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 election: Requesting vote from peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664) I0517 06:44:31.342218 8722 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" I0517 06:44:31.342624 8722 raft_consensus.cc:2756] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:31.342864 8722 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592: Snoozing failure detection for 2.068s (vote granted) I0517 06:44:31.348649 8722 raft_consensus.cc:2219] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10206d3db30a45a3a4c5a415d3ed2beb in term 1. I0517 06:44:31.349609 8282 leader_election.cc:379] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 election: Vote granted by peer 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7:45618) I0517 06:44:31.350154 8282 leader_election.cc:270] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:44:31.350631 8925 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb: Snoozing failure detection for 1.776s (election complete) I0517 06:44:31.350852 8925 raft_consensus.cc:2511] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:44:31.242599 8922 raft_consensus.cc:435] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:44:31.351805 8922 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Snoozing failure detection for 1.945s (starting election) I0517 06:44:31.352095 8922 raft_consensus.cc:457] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.352973 8925 raft_consensus.cc:571] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 1 LEADER]: Becoming Leader. State: Replica: 10206d3db30a45a3a4c5a415d3ed2beb, State: Running, Role: LEADER I0517 06:44:31.353549 8922 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3:43022) I0517 06:44:31.353608 8474 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Snoozing failure detection for 1.515s (vote granted) I0517 06:44:31.353838 8474 raft_consensus.cc:2219] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10206d3db30a45a3a4c5a415d3ed2beb in term 0. I0517 06:44:31.354384 8473 raft_consensus.cc:2194] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate 10206d3db30a45a3a4c5a415d3ed2beb for term 1 because replica is already servicing an update from a current leader or another vote. I0517 06:44:31.353651 8925 consensus_queue.cc:204] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.354826 8922 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7:45618) I0517 06:44:31.356631 8281 leader_election.cc:379] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 pre-election: Vote granted by peer dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438) I0517 06:44:31.357576 8281 leader_election.cc:393] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 election: Vote denied by peer dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438). Message: Service unavailable: T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate 10206d3db30a45a3a4c5a415d3ed2beb for term 1 because replica is already servicing an update from a current leader or another vote. I0517 06:44:31.357777 8922 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10206d3db30a45a3a4c5a415d3ed2beb (127.4.96.4:51856) I0517 06:44:31.360388 8922 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664) I0517 06:44:31.380313 7854 catalog_manager.cc:3686] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb reported cstate change: term changed from 0 to 1, leader changed from to 10206d3db30a45a3a4c5a415d3ed2beb (127.4.96.4). New cstate: current_term: 1 leader_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } health_report { overall_health: UNKNOWN } } } I0517 06:44:31.273349 8926 raft_consensus.cc:435] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:44:31.410487 8926 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Snoozing failure detection for 1.942s (starting election) I0517 06:44:31.410761 8926 raft_consensus.cc:457] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:31.412240 8926 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3:43022) I0517 06:44:31.412832 8926 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438) I0517 06:44:31.415110 8598 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Snoozing failure detection for 1.990s (vote granted) I0517 06:44:31.415364 8598 raft_consensus.cc:2219] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10206d3db30a45a3a4c5a415d3ed2beb in term 0. I0517 06:44:31.421805 8926 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7:45618) I0517 06:44:31.422319 8926 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 10206d3db30a45a3a4c5a415d3ed2beb (127.4.96.4:51856) I0517 06:44:31.433266 4480 external_mini_cluster.cc:482] 8 TS(s) registered with all masters I0517 06:44:31.437660 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5c25814d71724ad89c2b2965035156ab to finish bootstrapping I0517 06:44:31.453042 8350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "dafb8d2b9a584a74b41c05787477c5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" is_pre_election: true I0517 06:44:31.453671 8350 raft_consensus.cc:2180] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate dafb8d2b9a584a74b41c05787477c5c2 for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:44:31.455065 8404 leader_election.cc:393] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10206d3db30a45a3a4c5a415d3ed2beb (127.4.96.4:51856). Message: Invalid argument: T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate dafb8d2b9a584a74b41c05787477c5c2 for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:44:31.466233 8226 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "dafb8d2b9a584a74b41c05787477c5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3b5a36994a489eb39f15f8700fc089" is_pre_election: true I0517 06:44:31.466781 8226 raft_consensus.cc:2149] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dafb8d2b9a584a74b41c05787477c5c2 in current term 1: Already voted for candidate 10206d3db30a45a3a4c5a415d3ed2beb in this term. I0517 06:44:31.467938 8406 leader_election.cc:393] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3:43022). Message: Invalid argument: T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dafb8d2b9a584a74b41c05787477c5c2 in current term 1: Already voted for candidate 10206d3db30a45a3a4c5a415d3ed2beb in this term. I0517 06:44:31.475925 8722 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "dafb8d2b9a584a74b41c05787477c5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" is_pre_election: true I0517 06:44:31.476464 8722 raft_consensus.cc:2149] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dafb8d2b9a584a74b41c05787477c5c2 in current term 1: Already voted for candidate 10206d3db30a45a3a4c5a415d3ed2beb in this term. I0517 06:44:31.478235 8406 leader_election.cc:393] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7:45618). Message: Invalid argument: T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dafb8d2b9a584a74b41c05787477c5c2 in current term 1: Already voted for candidate 10206d3db30a45a3a4c5a415d3ed2beb in this term. I0517 06:44:31.478462 8406 leader_election.cc:270] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0517 06:44:31.478981 8922 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Snoozing failure detection for 1.612s (election complete) I0517 06:44:31.479045 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 580b6d55956e4c18bf9297327263345b to finish bootstrapping I0517 06:44:31.479179 8922 raft_consensus.cc:2756] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:31.483381 8922 raft_consensus.cc:2459] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0517 06:44:31.351074 8597 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc2430d5d05d4863b227c5cefc7ac460" I0517 06:44:31.484985 8597 raft_consensus.cc:2756] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:31.485285 8597 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Snoozing failure detection for 1.853s (vote granted) I0517 06:44:31.490694 8597 raft_consensus.cc:2219] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10206d3db30a45a3a4c5a415d3ed2beb in term 1. I0517 06:44:31.518668 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6a3b5a36994a489eb39f15f8700fc089 to finish bootstrapping I0517 06:44:31.538843 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 10206d3db30a45a3a4c5a415d3ed2beb to finish bootstrapping I0517 06:44:31.557862 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver dafb8d2b9a584a74b41c05787477c5c2 to finish bootstrapping W0517 06:44:31.390466 8642 tablet.cc:1678] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:31.591130 8284 leader_election.cc:379] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 pre-election: Vote granted by peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664) I0517 06:44:31.593837 8284 leader_election.cc:379] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [CANDIDATE]: Term 1 election: Vote granted by peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664) I0517 06:44:31.595656 8597 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "dafb8d2b9a584a74b41c05787477c5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc2430d5d05d4863b227c5cefc7ac460" is_pre_election: true I0517 06:44:31.596254 8597 raft_consensus.cc:2149] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dafb8d2b9a584a74b41c05787477c5c2 in current term 1: Already voted for candidate 10206d3db30a45a3a4c5a415d3ed2beb in this term. I0517 06:44:31.603066 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fc2430d5d05d4863b227c5cefc7ac460 to finish bootstrapping I0517 06:44:31.603101 8408 leader_election.cc:393] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 1 pre-election: Vote denied by peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664). Message: Invalid argument: T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dafb8d2b9a584a74b41c05787477c5c2 in current term 1: Already voted for candidate 10206d3db30a45a3a4c5a415d3ed2beb in this term. W0517 06:44:31.365255 8766 tablet.cc:1678] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:31.607515 8473 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "fc2430d5d05d4863b227c5cefc7ac460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dafb8d2b9a584a74b41c05787477c5c2" is_pre_election: true I0517 06:44:31.608012 8473 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Snoozing failure detection for 1.811s (vote granted) I0517 06:44:31.608223 8473 raft_consensus.cc:2219] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc2430d5d05d4863b227c5cefc7ac460 in term 1. I0517 06:44:31.609256 8529 leader_election.cc:379] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 1 pre-election: Vote granted by peer dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438) I0517 06:44:31.617645 8226 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "fc2430d5d05d4863b227c5cefc7ac460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3b5a36994a489eb39f15f8700fc089" is_pre_election: true I0517 06:44:31.618245 8226 raft_consensus.cc:2149] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fc2430d5d05d4863b227c5cefc7ac460 in current term 1: Already voted for candidate 10206d3db30a45a3a4c5a415d3ed2beb in this term. I0517 06:44:31.618266 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 51cb8e32a7ed4c7dbd5b79785556c592 to finish bootstrapping I0517 06:44:31.619355 8530 leader_election.cc:393] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3:43022). Message: Invalid argument: T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fc2430d5d05d4863b227c5cefc7ac460 in current term 1: Already voted for candidate 10206d3db30a45a3a4c5a415d3ed2beb in this term. I0517 06:44:31.621109 8350 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "fc2430d5d05d4863b227c5cefc7ac460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" is_pre_election: true I0517 06:44:31.621659 8350 raft_consensus.cc:2180] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate fc2430d5d05d4863b227c5cefc7ac460 for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:44:31.623562 8528 leader_election.cc:393] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 10206d3db30a45a3a4c5a415d3ed2beb (127.4.96.4:51856). Message: Invalid argument: T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate fc2430d5d05d4863b227c5cefc7ac460 for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:44:31.624821 8722 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "fc2430d5d05d4863b227c5cefc7ac460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" is_pre_election: true I0517 06:44:31.625345 8722 raft_consensus.cc:2149] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fc2430d5d05d4863b227c5cefc7ac460 in current term 1: Already voted for candidate 10206d3db30a45a3a4c5a415d3ed2beb in this term. I0517 06:44:31.626034 8530 leader_election.cc:393] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7:45618). Message: Invalid argument: T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fc2430d5d05d4863b227c5cefc7ac460 in current term 1: Already voted for candidate 10206d3db30a45a3a4c5a415d3ed2beb in this term. I0517 06:44:31.626214 8530 leader_election.cc:270] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0517 06:44:31.627413 8926 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Snoozing failure detection for 2.098s (election complete) I0517 06:44:31.627621 8926 raft_consensus.cc:2459] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0517 06:44:31.632412 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 92255276f9b144f3b14cc51ff4f450de to finish bootstrapping I0517 06:44:31.642829 4480 kudu-admin-test.cc:653] Waiting for Master to see the current replicas... I0517 06:44:31.645124 4480 kudu-admin-test.cc:660] Tablet locations: tablet_id: "b6bea5c6c58641f8ab69083b20103628" replicas { ts_info { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" rpc_addresses { host: "127.4.96.3" port: 43022 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" rpc_addresses { host: "127.4.96.5" port: 54438 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" rpc_addresses { host: "127.4.96.7" port: 45618 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" rpc_addresses { host: "127.4.96.4" port: 51856 } } role: LEADER } replicas { ts_info { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" rpc_addresses { host: "127.4.96.6" port: 47664 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0517 06:44:31.738466 8226 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P 6a3b5a36994a489eb39f15f8700fc089 [term 1 FOLLOWER]: Refusing update from remote peer 10206d3db30a45a3a4c5a415d3ed2beb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:31.739591 8925 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:31.781054 8722 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P 51cb8e32a7ed4c7dbd5b79785556c592 [term 1 FOLLOWER]: Refusing update from remote peer 10206d3db30a45a3a4c5a415d3ed2beb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:31.781931 8925 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:31.792165 8597 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 1 FOLLOWER]: Refusing update from remote peer 10206d3db30a45a3a4c5a415d3ed2beb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:31.794234 8954 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:31.797842 8473 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 1 FOLLOWER]: Refusing update from remote peer 10206d3db30a45a3a4c5a415d3ed2beb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:31.799324 8932 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P 10206d3db30a45a3a4c5a415d3ed2beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:31.827138 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 8644 W0517 06:44:31.846120 7836 connection.cc:512] client connection to 127.4.96.7:45618 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.7:45618): Connection reset by peer (error 104) W0517 06:44:31.846210 8282 connection.cc:512] client connection to 127.4.96.7:45618 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.7:45618): Connection reset by peer (error 104) W0517 06:44:31.847122 7838 connection.cc:512] server connection from 127.4.96.7:57193 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.7:57193): Connection reset by peer (error 104) W0517 06:44:31.847144 8406 connection.cc:512] client connection to 127.4.96.7:45618 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.7:45618): Connection reset by peer (error 104) W0517 06:44:31.848107 8530 connection.cc:512] client connection to 127.4.96.7:45618 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.7:45618): Connection reset by peer (error 104) W0517 06:44:31.849007 7825 connection.cc:512] client connection to 127.4.96.7:45618 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.7:45618): Connection reset by peer (error 104) I0517 06:44:31.850077 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 8148 W0517 06:44:31.868290 8282 connection.cc:512] client connection to 127.4.96.3:43022 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:43022): Connection reset by peer (error 104) W0517 06:44:31.868326 7834 connection.cc:512] server connection from 127.4.96.3:33222 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:33222): Connection reset by peer (error 104) W0517 06:44:31.868360 7836 connection.cc:512] client connection to 127.4.96.3:43022 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:43022): Connection reset by peer (error 104) W0517 06:44:31.869135 8530 connection.cc:512] client connection to 127.4.96.3:43022 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:43022): Connection reset by peer (error 104) W0517 06:44:31.869143 8406 connection.cc:512] client connection to 127.4.96.3:43022 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:43022): Connection reset by peer (error 104) W0517 06:44:31.869803 7825 connection.cc:512] client connection to 127.4.96.3:43022 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:43022): Connection reset by peer (error 104) I0517 06:44:31.870005 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 8272 W0517 06:44:31.897176 8404 connection.cc:512] client connection to 127.4.96.4:51856 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:51856): Connection reset by peer (error 104) W0517 06:44:31.897187 7834 connection.cc:512] client connection to 127.4.96.4:51856 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:51856): Connection reset by peer (error 104) W0517 06:44:31.897191 7836 connection.cc:512] server connection from 127.4.96.4:38185 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:38185): Connection reset by peer (error 104) W0517 06:44:31.898010 8408 connection.cc:512] server connection from 127.4.96.4:58023 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:58023): Connection reset by peer (error 104) W0517 06:44:31.898790 8528 connection.cc:512] client connection to 127.4.96.4:51856 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:51856): Connection reset by peer (error 104) W0517 06:44:31.898813 7825 connection.cc:512] client connection to 127.4.96.4:51856 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:51856): Connection reset by peer (error 104) W0517 06:44:31.898784 8529 connection.cc:512] server connection from 127.4.96.4:40549 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:40549): Connection reset by peer (error 104) W0517 06:44:31.899648 7821 connection.cc:512] client connection to 127.4.96.4:51856 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:51856): Connection reset by peer (error 104) I0517 06:44:31.900521 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 7826 I0517 06:44:31.917918 8889 heartbeater.cc:467] Master 127.4.96.62:54038 was elected leader, sending a full tablet report... W0517 06:44:31.924265 7907 connection.cc:512] client connection to 127.4.96.62:54038 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:54038): Connection reset by peer (error 104) W0517 06:44:31.924291 8777 connection.cc:512] client connection to 127.4.96.62:54038 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) I0517 06:44:31.924878 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:54038 --webserver_interface=127.4.96.62 --webserver_port=48617 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} W0517 06:44:31.925051 8777 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0517 06:44:31.925511 8406 connection.cc:512] server connection from 127.0.0.1:58513 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:58513): Connection reset by peer (error 104) W0517 06:44:31.930265 8032 connection.cc:512] client connection to 127.4.96.62:54038 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:54038): Connection reset by peer (error 104) W0517 06:44:31.930672 7825 connection.cc:512] client connection to 127.4.96.62:54038 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:54038): Connection reset by peer (error 104) W0517 06:44:31.931596 8405 connection.cc:512] client connection to 127.4.96.62:54038 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:54038): Connection reset by peer (error 104) W0517 06:44:31.932502 8529 connection.cc:512] server connection from 127.0.0.1:36421 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36421): Connection reset by peer (error 104) W0517 06:44:31.932910 8895 connection.cc:512] client connection to 127.4.96.62:54038 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:54038): Connection reset by peer (error 104) W0517 06:44:31.933182 8529 connection.cc:512] client connection to 127.4.96.62:54038 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:54038): Connection reset by peer (error 104) W0517 06:44:31.926111 8889 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:54038: Network error: Failed to send heartbeat to master: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:32.048118 8957 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:32.048422 8957 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:32.057384 8957 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:44:32.057505 8957 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:32.057602 8957 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:44:32.057713 8957 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:44:32.071475 8957 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:54038 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=48617 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:32.079115 8957 master_main.cc:80] Initializing master server... I0517 06:44:32.080557 8957 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 135319us I0517 06:44:32.081327 8957 fs_manager.cc:260] Metadata directory not provided I0517 06:44:32.081473 8957 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:32.087128 8957 fs_manager.cc:397] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s I0517 06:44:32.087765 8957 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:32.088178 8957 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:32.094784 8957 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0517 06:44:32.094909 8957 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "748e9fd9bc4b4d48808361d1ce5c7916" format_stamp: "Formatted at 2018-05-17 06:44:26 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:32.096078 8957 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:32.251720 8957 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:32.253466 8957 master_main.cc:83] Starting Master server... W0517 06:44:32.260823 8144 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:54038: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.96.62:54038: connect: Connection refused (error 111) I0517 06:44:32.294072 8957 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:54038 I0517 06:44:32.294937 8957 webserver.cc:173] Starting webserver on 127.4.96.62:48617 I0517 06:44:32.295044 8957 webserver.cc:184] Document root disabled I0517 06:44:32.296496 8957 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:48617/ I0517 06:44:32.297317 8957 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:44:32.306172 9019 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: Bootstrap starting. I0517 06:44:32.307502 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 8957 I0517 06:44:32.307893 4480 kudu-admin-test.cc:678] Forcing unsafe config change on tserver dafb8d2b9a584a74b41c05787477c5c2 I0517 06:44:32.342103 9019 log.cc:520] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: Log is configured to *not* fsync() on all Append() calls I0517 06:44:32.364348 9019 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:32.364940 9019 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: Bootstrap complete. I0517 06:44:32.382489 9019 raft_consensus.cc:303] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } I0517 06:44:32.384019 9019 raft_consensus.cc:604] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748e9fd9bc4b4d48808361d1ce5c7916, State: Initialized, Role: FOLLOWER I0517 06:44:32.384486 9019 consensus_queue.cc:227] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } I0517 06:44:32.384829 9019 raft_consensus.cc:343] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:44:32.385018 9019 raft_consensus.cc:435] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:44:32.385174 9019 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: Snoozing failure detection for 2.249s (starting election) I0517 06:44:32.385329 9019 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:44:32.389509 9019 raft_consensus.cc:457] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } I0517 06:44:32.389948 9019 leader_election.cc:270] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0517 06:44:32.399161 9022 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: Snoozing failure detection for 1.617s (election complete) I0517 06:44:32.399468 9022 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 2 FOLLOWER]: Leader election won for term 2 I0517 06:44:32.405292 9022 raft_consensus.cc:571] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [term 2 LEADER]: Becoming Leader. State: Replica: 748e9fd9bc4b4d48808361d1ce5c7916, State: Running, Role: LEADER I0517 06:44:32.405694 9022 consensus_queue.cc:204] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } I0517 06:44:32.410934 9019 sys_catalog.cc:335] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } } I0517 06:44:32.411247 9019 sys_catalog.cc:338] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: This master's current role is: LEADER W0517 06:44:32.460935 9020 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:44:32.411193 9023 sys_catalog.cc:335] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 748e9fd9bc4b4d48808361d1ce5c7916. Latest consensus state: current_term: 2 leader_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } } I0517 06:44:32.472743 9023 sys_catalog.cc:338] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: This master's current role is: LEADER I0517 06:44:32.413352 9024 sys_catalog.cc:335] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748e9fd9bc4b4d48808361d1ce5c7916" member_type: VOTER } } I0517 06:44:32.479223 9024 sys_catalog.cc:338] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: This master's current role is: LEADER I0517 06:44:32.479882 9019 sys_catalog.cc:422] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:44:32.480146 9026 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:44:32.481540 8957 master_main.cc:86] Master server successfully started. I0517 06:44:32.484966 9026 catalog_manager.cc:321] Loaded metadata for table TestTable [id=535082618aca4f96896c631f72424763] I0517 06:44:32.487107 9026 catalog_manager.cc:375] Loaded metadata for tablet b6bea5c6c58641f8ab69083b20103628 (table TestTable [id=535082618aca4f96896c631f72424763]) I0517 06:44:32.487504 9026 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:44:32.490790 9026 catalog_manager.cc:1021] Loading token signing keys... I0517 06:44:32.492220 9026 catalog_manager.cc:3973] T 00000000000000000000000000000000 P 748e9fd9bc4b4d48808361d1ce5c7916: Loaded TSK: 0 I0517 06:44:32.693667 8986 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "5c25814d71724ad89c2b2965035156ab" instance_seqno: 1526539467602539) as {username='slave'} at 127.4.96.1:41607; Asking this server to re-register. I0517 06:44:32.695168 8019 heartbeater.cc:426] Registering TS with master... I0517 06:44:32.695503 8019 heartbeater.cc:475] Master 127.4.96.62:54038 requested a full tablet report, sending... I0517 06:44:32.696952 8986 ts_manager.cc:81] Registered new tserver with Master: 5c25814d71724ad89c2b2965035156ab (127.4.96.1:43055) I0517 06:44:32.716976 8473 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "dafb8d2b9a584a74b41c05787477c5c2" tablet_id: "b6bea5c6c58641f8ab69083b20103628" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" } } from {username='slave'} at 127.0.0.1:58528 W0517 06:44:32.717747 8473 raft_consensus.cc:1990] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 1 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } }NEW CONFIG: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } unsafe_config_change: true I0517 06:44:32.718504 8473 raft_consensus.cc:2756] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:44:32.751052 8986 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" instance_seqno: 1526539469440756) as {username='slave'} at 127.4.96.5:43556; Asking this server to re-register. I0517 06:44:32.751935 8517 heartbeater.cc:426] Registering TS with master... I0517 06:44:32.752169 8517 heartbeater.cc:475] Master 127.4.96.62:54038 requested a full tablet report, sending... I0517 06:44:32.753523 8986 ts_manager.cc:81] Registered new tserver with Master: dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438) I0517 06:44:32.845734 8986 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" instance_seqno: 1526539469864995) as {username='slave'} at 127.4.96.6:45411; Asking this server to re-register. I0517 06:44:32.846616 8641 heartbeater.cc:426] Registering TS with master... I0517 06:44:32.846860 8641 heartbeater.cc:475] Master 127.4.96.62:54038 requested a full tablet report, sending... I0517 06:44:32.848129 8986 ts_manager.cc:81] Registered new tserver with Master: fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664) I0517 06:44:32.948782 8889 heartbeater.cc:346] Connected to a master server at 127.4.96.62:54038 I0517 06:44:32.949053 8889 heartbeater.cc:467] Master 127.4.96.62:54038 was elected leader, sending a full tablet report... I0517 06:44:32.950217 8986 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" instance_seqno: 1526539470845763) as {username='slave'} at 127.4.96.8:55213; Asking this server to re-register. I0517 06:44:32.951089 8889 heartbeater.cc:426] Registering TS with master... I0517 06:44:32.951326 8889 heartbeater.cc:475] Master 127.4.96.62:54038 requested a full tablet report, sending... I0517 06:44:32.952431 8986 ts_manager.cc:81] Registered new tserver with Master: 92255276f9b144f3b14cc51ff4f450de (127.4.96.8:55767) I0517 06:44:33.278293 8144 heartbeater.cc:346] Connected to a master server at 127.4.96.62:54038 I0517 06:44:33.279556 8986 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "580b6d55956e4c18bf9297327263345b" instance_seqno: 1526539468166664) as {username='slave'} at 127.4.96.2:56709; Asking this server to re-register. I0517 06:44:33.280422 8144 heartbeater.cc:426] Registering TS with master... I0517 06:44:33.280661 8144 heartbeater.cc:475] Master 127.4.96.62:54038 requested a full tablet report, sending... I0517 06:44:33.281759 8986 ts_manager.cc:81] Registered new tserver with Master: 580b6d55956e4c18bf9297327263345b (127.4.96.2:33256) I0517 06:44:33.302618 9052 raft_consensus.cc:435] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10206d3db30a45a3a4c5a415d3ed2beb) I0517 06:44:33.302899 9052 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Snoozing failure detection for 2.162s (starting election) I0517 06:44:33.303148 9052 raft_consensus.cc:457] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } I0517 06:44:33.304333 9052 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3:43022) I0517 06:44:33.304736 9052 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438) I0517 06:44:33.305485 9052 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7:45618) W0517 06:44:33.305660 8530 leader_election.cc:299] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3:43022): Network error: Client connection negotiation failed: client connection to 127.4.96.3:43022: connect: Connection refused (error 111) I0517 06:44:33.305960 9052 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 10206d3db30a45a3a4c5a415d3ed2beb (127.4.96.4:51856) I0517 06:44:33.306100 8473 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "fc2430d5d05d4863b227c5cefc7ac460" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dafb8d2b9a584a74b41c05787477c5c2" is_pre_election: true I0517 06:44:33.306649 8473 raft_consensus.cc:2180] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fc2430d5d05d4863b227c5cefc7ac460 for term 2 because replica is either leader or believes a valid leader to be alive. I0517 06:44:33.307670 8529 leader_election.cc:393] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 2 pre-election: Vote denied by peer dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438). Message: Invalid argument: T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fc2430d5d05d4863b227c5cefc7ac460 for term 2 because replica is either leader or believes a valid leader to be alive. W0517 06:44:33.308068 8530 leader_election.cc:299] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7:45618): Network error: Client connection negotiation failed: client connection to 127.4.96.7:45618: connect: Connection refused (error 111) I0517 06:44:33.308256 8530 leader_election.cc:270] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0517 06:44:33.312578 9052 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Snoozing failure detection for 1.561s (election complete) I0517 06:44:33.312788 9052 raft_consensus.cc:2756] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 1 FOLLOWER]: Advancing to term 2 W0517 06:44:33.314568 8528 leader_election.cc:299] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10206d3db30a45a3a4c5a415d3ed2beb (127.4.96.4:51856): Network error: Client connection negotiation failed: client connection to 127.4.96.4:51856: connect: Connection refused (error 111) I0517 06:44:33.317580 9052 raft_consensus.cc:2459] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0517 06:44:34.232017 9057 raft_consensus.cc:435] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0517 06:44:34.232246 9057 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Snoozing failure detection for 1.919s (starting election) I0517 06:44:34.232426 9057 raft_consensus.cc:457] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } unsafe_config_change: true I0517 06:44:34.232967 9057 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664) I0517 06:44:34.234064 8597 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "dafb8d2b9a584a74b41c05787477c5c2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fc2430d5d05d4863b227c5cefc7ac460" is_pre_election: true I0517 06:44:34.234402 8597 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Snoozing failure detection for 2.302s (vote granted) I0517 06:44:34.234547 8597 raft_consensus.cc:2219] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dafb8d2b9a584a74b41c05787477c5c2 in term 2. I0517 06:44:34.235244 8408 leader_election.cc:379] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 3 pre-election: Vote granted by peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664) I0517 06:44:34.235409 8408 leader_election.cc:270] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0517 06:44:34.235759 9057 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Snoozing failure detection for 1.977s (election complete) I0517 06:44:34.235963 9057 raft_consensus.cc:2511] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0517 06:44:34.236086 9057 raft_consensus.cc:435] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0517 06:44:34.236210 9057 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Snoozing failure detection for 1.906s (starting election) I0517 06:44:34.236323 9057 raft_consensus.cc:2756] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 2 FOLLOWER]: Advancing to term 3 I0517 06:44:34.239235 9057 raft_consensus.cc:457] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } unsafe_config_change: true I0517 06:44:34.239732 9057 leader_election.cc:243] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 3 election: Requesting vote from peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664) I0517 06:44:34.240684 8597 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b6bea5c6c58641f8ab69083b20103628" candidate_uuid: "dafb8d2b9a584a74b41c05787477c5c2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fc2430d5d05d4863b227c5cefc7ac460" I0517 06:44:34.240973 8597 raft_consensus.cc:2756] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 2 FOLLOWER]: Advancing to term 3 I0517 06:44:34.241129 8597 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460: Snoozing failure detection for 3.076s (vote granted) I0517 06:44:34.243702 8597 raft_consensus.cc:2219] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dafb8d2b9a584a74b41c05787477c5c2 in term 3. I0517 06:44:34.244302 8408 leader_election.cc:379] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 3 election: Vote granted by peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664) I0517 06:44:34.244457 8408 leader_election.cc:270] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0517 06:44:34.244774 9057 raft_consensus.cc:2704] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Snoozing failure detection for 2.209s (election complete) I0517 06:44:34.244949 9057 raft_consensus.cc:2511] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 3 FOLLOWER]: Leader election won for term 3 I0517 06:44:34.245805 9057 raft_consensus.cc:571] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 3 LEADER]: Becoming Leader. State: Replica: dafb8d2b9a584a74b41c05787477c5c2, State: Running, Role: LEADER I0517 06:44:34.246224 9057 consensus_queue.cc:204] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } unsafe_config_change: true I0517 06:44:34.249711 8986 catalog_manager.cc:3686] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 reported cstate change: term changed from 1 to 3, leader changed from 10206d3db30a45a3a4c5a415d3ed2beb (127.4.96.4) to dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5), now has a pending config: VOTER dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5), VOTER fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6). New cstate: current_term: 3 leader_uuid: "dafb8d2b9a584a74b41c05787477c5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3b5a36994a489eb39f15f8700fc089" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 43022 } } peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51cb8e32a7ed4c7dbd5b79785556c592" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 45618 } } peers { permanent_uuid: "10206d3db30a45a3a4c5a415d3ed2beb" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 51856 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } unsafe_config_change: true } I0517 06:44:34.654703 8597 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch) I0517 06:44:34.655658 9057 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s I0517 06:44:34.661370 9058 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 3 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 10206d3db30a45a3a4c5a415d3ed2beb (127.4.96.4) evicted, VOTER 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7) evicted, VOTER 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } unsafe_config_change: true } I0517 06:44:34.662801 8597 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 10206d3db30a45a3a4c5a415d3ed2beb (127.4.96.4) evicted, VOTER 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7) evicted, VOTER 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } unsafe_config_change: true } I0517 06:44:34.668215 8986 catalog_manager.cc:3686] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 reported cstate change: config changed from index -1 to 2, VOTER 10206d3db30a45a3a4c5a415d3ed2beb (127.4.96.4) evicted, VOTER 51cb8e32a7ed4c7dbd5b79785556c592 (127.4.96.7) evicted, VOTER 6a3b5a36994a489eb39f15f8700fc089 (127.4.96.3) evicted, no longer has a pending config: VOTER dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5), VOTER fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6). New cstate: current_term: 3 leader_uuid: "dafb8d2b9a584a74b41c05787477c5c2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:44:34.672577 8986 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089 with a delay of 26 ms (attempt = 1) W0517 06:44:34.672755 8986 catalog_manager.cc:3845] Failed to send b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089: Not found: failed to reset TS proxy: Could not find TS for UUID 6a3b5a36994a489eb39f15f8700fc089 I0517 06:44:34.672920 8986 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592 with a delay of 21 ms (attempt = 1) W0517 06:44:34.673049 8986 catalog_manager.cc:3845] Failed to send b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592: Not found: failed to reset TS proxy: Could not find TS for UUID 51cb8e32a7ed4c7dbd5b79785556c592 I0517 06:44:34.673319 8986 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb with a delay of 34 ms (attempt = 1) W0517 06:44:34.673519 8986 catalog_manager.cc:3845] Failed to send b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb: Not found: failed to reset TS proxy: Could not find TS for UUID 10206d3db30a45a3a4c5a415d3ed2beb I0517 06:44:34.674111 8986 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet b6bea5c6c58641f8ab69083b20103628 (attempt 1) I0517 06:44:34.678002 8985 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet b6bea5c6c58641f8ab69083b20103628 (attempt 1) I0517 06:44:34.685290 8473 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664): Closing peer: fc2430d5d05d4863b227c5cefc7ac460 I0517 06:44:34.685684 8473 consensus_queue.cc:204] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:34.688539 8597 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch) I0517 06:44:34.689167 8966 catalog_manager.cc:3257] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b6bea5c6c58641f8ab69083b20103628 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER failed with leader dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } unsafe_config_change: true. Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:34.689510 8966 catalog_manager.cc:2791] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b6bea5c6c58641f8ab69083b20103628 with cas_config_opid_index 2 with a delay of 58 ms (attempt = 1) I0517 06:44:34.689656 9057 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0517 06:44:34.694470 8967 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592 with a delay of 64 ms (attempt = 2) W0517 06:44:34.694707 8967 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 51cb8e32a7ed4c7dbd5b79785556c592 I0517 06:44:34.693588 9058 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER 580b6d55956e4c18bf9297327263345b (127.4.96.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:34.694864 8597 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER 580b6d55956e4c18bf9297327263345b (127.4.96.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:34.698990 8965 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089 with a delay of 78 ms (attempt = 2) W0517 06:44:34.699225 8965 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 6a3b5a36994a489eb39f15f8700fc089 I0517 06:44:34.700160 8966 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b6bea5c6c58641f8ab69083b20103628 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) W0517 06:44:34.701843 8405 consensus_peers.cc:434] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 580b6d55956e4c18bf9297327263345b (127.4.96.2:33256): Couldn't send request to peer 580b6d55956e4c18bf9297327263345b for tablet b6bea5c6c58641f8ab69083b20103628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6bea5c6c58641f8ab69083b20103628. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:34.702394 8985 catalog_manager.cc:3686] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 reported cstate change: config changed from index 2 to 4, NON_VOTER 580b6d55956e4c18bf9297327263345b (127.4.96.2) added. New cstate: current_term: 3 leader_uuid: "dafb8d2b9a584a74b41c05787477c5c2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:34.706966 8984 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet b6bea5c6c58641f8ab69083b20103628 (attempt 1) I0517 06:44:34.707897 8969 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb with a delay of 61 ms (attempt = 2) W0517 06:44:34.708102 8969 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb failed: Not found: failed to reset TS proxy: Could not find TS for UUID 10206d3db30a45a3a4c5a415d3ed2beb I0517 06:44:34.708704 8474 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 580b6d55956e4c18bf9297327263345b (127.4.96.2:33256): Closing peer: 580b6d55956e4c18bf9297327263345b I0517 06:44:34.708905 8474 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664): Closing peer: fc2430d5d05d4863b227c5cefc7ac460 I0517 06:44:34.709441 8474 consensus_queue.cc:204] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:34.710220 8984 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet b6bea5c6c58641f8ab69083b20103628 (attempt 1) I0517 06:44:34.713302 8966 catalog_manager.cc:3257] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b6bea5c6c58641f8ab69083b20103628 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER failed with leader dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } unsafe_config_change: true. Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:34.713608 8966 catalog_manager.cc:2791] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b6bea5c6c58641f8ab69083b20103628 with cas_config_opid_index 4 with a delay of 37 ms (attempt = 1) W0517 06:44:34.714179 8405 consensus_peers.cc:434] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 580b6d55956e4c18bf9297327263345b (127.4.96.2:33256): Couldn't send request to peer 580b6d55956e4c18bf9297327263345b for tablet b6bea5c6c58641f8ab69083b20103628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6bea5c6c58641f8ab69083b20103628. Retrying in the next heartbeat period. Already tried 1 times. W0517 06:44:34.721631 8404 consensus_peers.cc:434] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 5c25814d71724ad89c2b2965035156ab (127.4.96.1:43055): Couldn't send request to peer 5c25814d71724ad89c2b2965035156ab for tablet b6bea5c6c58641f8ab69083b20103628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6bea5c6c58641f8ab69083b20103628. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:34.723683 8597 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0517 06:44:34.724474 9058 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0517 06:44:34.727434 9071 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, NON_VOTER 5c25814d71724ad89c2b2965035156ab (127.4.96.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:34.728322 8597 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, NON_VOTER 5c25814d71724ad89c2b2965035156ab (127.4.96.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:34.731348 8966 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b6bea5c6c58641f8ab69083b20103628 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0517 06:44:34.733438 8984 catalog_manager.cc:3686] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 reported cstate change: config changed from index 4 to 5, NON_VOTER 5c25814d71724ad89c2b2965035156ab (127.4.96.1) added. New cstate: current_term: 3 leader_uuid: "dafb8d2b9a584a74b41c05787477c5c2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:34.738795 8985 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet b6bea5c6c58641f8ab69083b20103628 (attempt 1) I0517 06:44:34.740100 8474 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 5c25814d71724ad89c2b2965035156ab (127.4.96.1:43055): Closing peer: 5c25814d71724ad89c2b2965035156ab I0517 06:44:34.740247 8474 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 580b6d55956e4c18bf9297327263345b (127.4.96.2:33256): Closing peer: 580b6d55956e4c18bf9297327263345b I0517 06:44:34.740371 8474 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664): Closing peer: fc2430d5d05d4863b227c5cefc7ac460 I0517 06:44:34.740761 8474 consensus_queue.cc:204] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true W0517 06:44:34.744652 8404 consensus_peers.cc:434] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 5c25814d71724ad89c2b2965035156ab (127.4.96.1:43055): Couldn't send request to peer 5c25814d71724ad89c2b2965035156ab for tablet b6bea5c6c58641f8ab69083b20103628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6bea5c6c58641f8ab69083b20103628. Retrying in the next heartbeat period. Already tried 1 times. W0517 06:44:34.745473 8405 consensus_peers.cc:434] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 580b6d55956e4c18bf9297327263345b (127.4.96.2:33256): Couldn't send request to peer 580b6d55956e4c18bf9297327263345b for tablet b6bea5c6c58641f8ab69083b20103628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6bea5c6c58641f8ab69083b20103628. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:34.746301 8597 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch) I0517 06:44:34.747274 9073 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0517 06:44:34.748731 8966 catalog_manager.cc:3273] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b6bea5c6c58641f8ab69083b20103628 with cas_config_opid_index 2: aborting the task: latest config opid_index 5; task opid_index 2 I0517 06:44:34.751382 8966 catalog_manager.cc:3273] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b6bea5c6c58641f8ab69083b20103628 with cas_config_opid_index 4: aborting the task: latest config opid_index 5; task opid_index 4 I0517 06:44:34.750953 9071 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER 92255276f9b144f3b14cc51ff4f450de (127.4.96.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:34.752108 8597 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER 92255276f9b144f3b14cc51ff4f450de (127.4.96.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:34.754935 8966 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b6bea5c6c58641f8ab69083b20103628 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) W0517 06:44:34.757475 8406 consensus_peers.cc:434] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 92255276f9b144f3b14cc51ff4f450de (127.4.96.8:55767): Couldn't send request to peer 92255276f9b144f3b14cc51ff4f450de for tablet b6bea5c6c58641f8ab69083b20103628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6bea5c6c58641f8ab69083b20103628. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:34.757125 8985 catalog_manager.cc:3686] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 reported cstate change: config changed from index 5 to 6, NON_VOTER 92255276f9b144f3b14cc51ff4f450de (127.4.96.8) added. New cstate: current_term: 3 leader_uuid: "dafb8d2b9a584a74b41c05787477c5c2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0517 06:44:34.759552 8967 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592 with a delay of 84 ms (attempt = 3) W0517 06:44:34.759723 8967 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 51cb8e32a7ed4c7dbd5b79785556c592 I0517 06:44:34.769637 8969 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb with a delay of 98 ms (attempt = 3) W0517 06:44:34.769807 8969 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb failed: Not found: failed to reset TS proxy: Could not find TS for UUID 10206d3db30a45a3a4c5a415d3ed2beb I0517 06:44:34.777889 8965 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089 with a delay of 86 ms (attempt = 3) W0517 06:44:34.778048 8965 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 6a3b5a36994a489eb39f15f8700fc089 I0517 06:44:34.844355 8967 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592 with a delay of 155 ms (attempt = 4) W0517 06:44:34.844521 8967 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 51cb8e32a7ed4c7dbd5b79785556c592 I0517 06:44:34.864609 8965 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089 with a delay of 162 ms (attempt = 4) W0517 06:44:34.864754 8965 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 6a3b5a36994a489eb39f15f8700fc089 I0517 06:44:34.868405 8969 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb with a delay of 168 ms (attempt = 4) W0517 06:44:34.868547 8969 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb failed: Not found: failed to reset TS proxy: Could not find TS for UUID 10206d3db30a45a3a4c5a415d3ed2beb I0517 06:44:35.000166 8967 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592 with a delay of 289 ms (attempt = 5) W0517 06:44:35.000361 8967 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 51cb8e32a7ed4c7dbd5b79785556c592 I0517 06:44:35.027577 8965 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089 with a delay of 278 ms (attempt = 5) W0517 06:44:35.027736 8965 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 6a3b5a36994a489eb39f15f8700fc089 I0517 06:44:35.037462 8969 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb with a delay of 304 ms (attempt = 5) W0517 06:44:35.037613 8969 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb failed: Not found: failed to reset TS proxy: Could not find TS for UUID 10206d3db30a45a3a4c5a415d3ed2beb I0517 06:44:35.205579 9075 ts_tablet_manager.cc:669] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: Initiating tablet copy from peer dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438) I0517 06:44:35.205853 9075 tablet_copy_client.cc:239] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.5:54438 I0517 06:44:35.215135 8494 tablet_copy_service.cc:133] P dafb8d2b9a584a74b41c05787477c5c2: Received BeginTabletCopySession request for tablet b6bea5c6c58641f8ab69083b20103628 from peer 92255276f9b144f3b14cc51ff4f450de ({username='slave'} at 127.4.96.8:40224) I0517 06:44:35.215381 8494 tablet_copy_service.cc:154] P dafb8d2b9a584a74b41c05787477c5c2: Beginning new tablet copy session on tablet b6bea5c6c58641f8ab69083b20103628 from peer 92255276f9b144f3b14cc51ff4f450de at {username='slave'} at 127.4.96.8:40224: session id = 92255276f9b144f3b14cc51ff4f450de-b6bea5c6c58641f8ab69083b20103628 I0517 06:44:35.218667 8494 tablet_copy_source_session.cc:202] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:44:35.221282 9075 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b6bea5c6c58641f8ab69083b20103628. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:35.233261 9075 ts_tablet_manager.cc:1173] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:44:35.233671 9075 tablet_copy_client.cc:565] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: tablet copy: Starting download of 0 data blocks... I0517 06:44:35.234017 9075 tablet_copy_client.cc:528] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: tablet copy: Starting download of 1 WAL segments... I0517 06:44:35.236207 9075 tablet_copy_client.cc:404] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:44:35.236322 9078 ts_tablet_manager.cc:669] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: Initiating tablet copy from peer dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438) I0517 06:44:35.236578 9078 tablet_copy_client.cc:239] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.5:54438 I0517 06:44:35.238991 9075 ts_tablet_manager.cc:1028] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: Bootstrapping tablet I0517 06:44:35.239342 9075 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: Bootstrap starting. I0517 06:44:35.244750 9075 log.cc:520] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: Log is configured to *not* fsync() on all Append() calls I0517 06:44:35.253427 9075 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:35.253492 8494 tablet_copy_service.cc:133] P dafb8d2b9a584a74b41c05787477c5c2: Received BeginTabletCopySession request for tablet b6bea5c6c58641f8ab69083b20103628 from peer 5c25814d71724ad89c2b2965035156ab ({username='slave'} at 127.4.96.1:36759) I0517 06:44:35.253741 8494 tablet_copy_service.cc:154] P dafb8d2b9a584a74b41c05787477c5c2: Beginning new tablet copy session on tablet b6bea5c6c58641f8ab69083b20103628 from peer 5c25814d71724ad89c2b2965035156ab at {username='slave'} at 127.4.96.1:36759: session id = 5c25814d71724ad89c2b2965035156ab-b6bea5c6c58641f8ab69083b20103628 I0517 06:44:35.254015 9075 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: Bootstrap complete. I0517 06:44:35.254447 9075 ts_tablet_manager.cc:1045] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: Time spent bootstrapping tablet: real 0.015s user 0.016s sys 0.000s I0517 06:44:35.256945 8494 tablet_copy_source_session.cc:202] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:44:35.259498 9078 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b6bea5c6c58641f8ab69083b20103628. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:35.269951 9075 raft_consensus.cc:303] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:35.270956 9075 raft_consensus.cc:604] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 92255276f9b144f3b14cc51ff4f450de, State: Initialized, Role: LEARNER I0517 06:44:35.271414 9075 consensus_queue.cc:227] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:35.276347 9078 ts_tablet_manager.cc:1173] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:44:35.277312 9078 tablet_copy_client.cc:565] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: tablet copy: Starting download of 0 data blocks... I0517 06:44:35.277623 9078 tablet_copy_client.cc:528] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: tablet copy: Starting download of 1 WAL segments... I0517 06:44:35.280413 9078 tablet_copy_client.cc:404] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:44:35.281983 9075 ts_tablet_manager.cc:1073] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: Time spent starting tablet: real 0.027s user 0.016s sys 0.008s I0517 06:44:35.282631 9081 ts_tablet_manager.cc:669] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: Initiating tablet copy from peer dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5:54438) I0517 06:44:35.283004 9081 tablet_copy_client.cc:239] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.5:54438 I0517 06:44:35.287783 9078 ts_tablet_manager.cc:1028] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: Bootstrapping tablet I0517 06:44:35.287906 8494 tablet_copy_service.cc:323] P dafb8d2b9a584a74b41c05787477c5c2: Request end of tablet copy session 92255276f9b144f3b14cc51ff4f450de-b6bea5c6c58641f8ab69083b20103628 received from {username='slave'} at 127.4.96.8:40224 I0517 06:44:35.288118 8494 tablet_copy_service.cc:411] P dafb8d2b9a584a74b41c05787477c5c2: Ending tablet copy session 92255276f9b144f3b14cc51ff4f450de-b6bea5c6c58641f8ab69083b20103628 on tablet b6bea5c6c58641f8ab69083b20103628 with peer 92255276f9b144f3b14cc51ff4f450de I0517 06:44:35.288254 9078 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: Bootstrap starting. I0517 06:44:35.290516 8967 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592 with a delay of 542 ms (attempt = 6) W0517 06:44:35.290691 8967 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 51cb8e32a7ed4c7dbd5b79785556c592 I0517 06:44:35.295594 9078 log.cc:520] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: Log is configured to *not* fsync() on all Append() calls I0517 06:44:35.304131 9078 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:35.304713 9078 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: Bootstrap complete. I0517 06:44:35.305119 9078 ts_tablet_manager.cc:1045] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s I0517 06:44:35.305958 8494 tablet_copy_service.cc:133] P dafb8d2b9a584a74b41c05787477c5c2: Received BeginTabletCopySession request for tablet b6bea5c6c58641f8ab69083b20103628 from peer 580b6d55956e4c18bf9297327263345b ({username='slave'} at 127.4.96.2:41345) I0517 06:44:35.306165 8494 tablet_copy_service.cc:154] P dafb8d2b9a584a74b41c05787477c5c2: Beginning new tablet copy session on tablet b6bea5c6c58641f8ab69083b20103628 from peer 580b6d55956e4c18bf9297327263345b at {username='slave'} at 127.4.96.2:41345: session id = 580b6d55956e4c18bf9297327263345b-b6bea5c6c58641f8ab69083b20103628 I0517 06:44:35.306701 8965 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089 with a delay of 525 ms (attempt = 6) W0517 06:44:35.306917 8965 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 6a3b5a36994a489eb39f15f8700fc089 I0517 06:44:35.309388 8494 tablet_copy_source_session.cc:202] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:44:35.312085 9081 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b6bea5c6c58641f8ab69083b20103628. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:35.317096 9078 raft_consensus.cc:303] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:35.318080 9078 raft_consensus.cc:604] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 5c25814d71724ad89c2b2965035156ab, State: Initialized, Role: LEARNER I0517 06:44:35.318547 9078 consensus_queue.cc:227] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:35.321220 9078 ts_tablet_manager.cc:1073] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: Time spent starting tablet: real 0.016s user 0.012s sys 0.008s I0517 06:44:35.322279 8494 tablet_copy_service.cc:323] P dafb8d2b9a584a74b41c05787477c5c2: Request end of tablet copy session 5c25814d71724ad89c2b2965035156ab-b6bea5c6c58641f8ab69083b20103628 received from {username='slave'} at 127.4.96.1:36759 I0517 06:44:35.322525 8494 tablet_copy_service.cc:411] P dafb8d2b9a584a74b41c05787477c5c2: Ending tablet copy session 5c25814d71724ad89c2b2965035156ab-b6bea5c6c58641f8ab69083b20103628 on tablet b6bea5c6c58641f8ab69083b20103628 with peer 5c25814d71724ad89c2b2965035156ab I0517 06:44:35.327497 9081 ts_tablet_manager.cc:1173] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:44:35.327841 9081 tablet_copy_client.cc:565] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: tablet copy: Starting download of 0 data blocks... I0517 06:44:35.328058 9081 tablet_copy_client.cc:528] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: tablet copy: Starting download of 1 WAL segments... I0517 06:44:35.329955 9081 tablet_copy_client.cc:404] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:44:35.332628 9081 ts_tablet_manager.cc:1028] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: Bootstrapping tablet I0517 06:44:35.332986 9081 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: Bootstrap starting. I0517 06:44:35.338269 9081 log.cc:520] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: Log is configured to *not* fsync() on all Append() calls I0517 06:44:35.342419 8969 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb with a delay of 559 ms (attempt = 6) W0517 06:44:35.342653 8969 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb failed: Not found: failed to reset TS proxy: Could not find TS for UUID 10206d3db30a45a3a4c5a415d3ed2beb I0517 06:44:35.344923 9081 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:35.345372 9081 tablet_bootstrap.cc:436] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: Bootstrap complete. I0517 06:44:35.345677 9081 ts_tablet_manager.cc:1045] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s I0517 06:44:35.354940 9081 raft_consensus.cc:303] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:35.355593 9081 raft_consensus.cc:604] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 580b6d55956e4c18bf9297327263345b, State: Initialized, Role: LEARNER I0517 06:44:35.355919 9081 consensus_queue.cc:227] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: true } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:35.357847 9081 ts_tablet_manager.cc:1073] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: Time spent starting tablet: real 0.012s user 0.012s sys 0.004s I0517 06:44:35.358809 8494 tablet_copy_service.cc:323] P dafb8d2b9a584a74b41c05787477c5c2: Request end of tablet copy session 580b6d55956e4c18bf9297327263345b-b6bea5c6c58641f8ab69083b20103628 received from {username='slave'} at 127.4.96.2:41345 I0517 06:44:35.358981 8494 tablet_copy_service.cc:411] P dafb8d2b9a584a74b41c05787477c5c2: Ending tablet copy session 580b6d55956e4c18bf9297327263345b-b6bea5c6c58641f8ab69083b20103628 on tablet b6bea5c6c58641f8ab69083b20103628 with peer 580b6d55956e4c18bf9297327263345b W0517 06:44:35.386274 8890 tablet.cc:1678] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:44:35.387632 8020 tablet.cc:1678] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:44:35.468020 8145 tablet.cc:1678] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:35.691431 8846 raft_consensus.cc:1002] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0517 06:44:35.750705 7976 raft_consensus.cc:1002] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0517 06:44:35.789083 8101 raft_consensus.cc:1002] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0517 06:44:35.832231 8965 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089 with a delay of 1024 ms (attempt = 7) W0517 06:44:35.832387 8965 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=6a3b5a36994a489eb39f15f8700fc089 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 6a3b5a36994a489eb39f15f8700fc089 I0517 06:44:35.833117 8967 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592 with a delay of 1054 ms (attempt = 7) W0517 06:44:35.833261 8967 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=51cb8e32a7ed4c7dbd5b79785556c592 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 51cb8e32a7ed4c7dbd5b79785556c592 I0517 06:44:35.902868 8969 catalog_manager.cc:2791] Scheduling retry of b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb with a delay of 1047 ms (attempt = 7) W0517 06:44:35.903038 8969 catalog_manager.cc:2810] Async tablet task b6bea5c6c58641f8ab69083b20103628 Delete Tablet RPC for TS=10206d3db30a45a3a4c5a415d3ed2beb failed: Not found: failed to reset TS proxy: Could not find TS for UUID 10206d3db30a45a3a4c5a415d3ed2beb I0517 06:44:36.220041 9089 raft_consensus.cc:866] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: attempting to promote NON_VOTER 580b6d55956e4c18bf9297327263345b to VOTER I0517 06:44:36.220871 9089 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 5c25814d71724ad89c2b2965035156ab (127.4.96.1:43055): Closing peer: 5c25814d71724ad89c2b2965035156ab I0517 06:44:36.221058 9089 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 580b6d55956e4c18bf9297327263345b (127.4.96.2:33256): Closing peer: 580b6d55956e4c18bf9297327263345b I0517 06:44:36.221199 9089 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 92255276f9b144f3b14cc51ff4f450de (127.4.96.8:55767): Closing peer: 92255276f9b144f3b14cc51ff4f450de I0517 06:44:36.221350 9089 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664): Closing peer: fc2430d5d05d4863b227c5cefc7ac460 I0517 06:44:36.221951 9089 consensus_queue.cc:204] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:36.226711 8101 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b [term 3 LEARNER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:44:36.226714 7976 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab [term 3 LEARNER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:44:36.227254 8846 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de [term 3 LEARNER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:44:36.227538 8597 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:44:36.228037 9073 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:44:36.228497 9088 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:44:36.229874 9073 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:44:36.230356 9091 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:44:36.244287 9088 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 580b6d55956e4c18bf9297327263345b (127.4.96.2) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:36.245820 8597 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 580b6d55956e4c18bf9297327263345b (127.4.96.2) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:36.251585 8101 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 580b6d55956e4c18bf9297327263345b (127.4.96.2) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:36.255103 8984 catalog_manager.cc:3686] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 reported cstate change: config changed from index 6 to 7, 580b6d55956e4c18bf9297327263345b (127.4.96.2) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "dafb8d2b9a584a74b41c05787477c5c2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0517 06:44:36.260857 9073 raft_consensus.cc:866] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: attempting to promote NON_VOTER 5c25814d71724ad89c2b2965035156ab to VOTER I0517 06:44:36.261693 9073 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 5c25814d71724ad89c2b2965035156ab (127.4.96.1:43055): Closing peer: 5c25814d71724ad89c2b2965035156ab I0517 06:44:36.261886 9073 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 580b6d55956e4c18bf9297327263345b (127.4.96.2:33256): Closing peer: 580b6d55956e4c18bf9297327263345b I0517 06:44:36.262058 9073 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 92255276f9b144f3b14cc51ff4f450de (127.4.96.8:55767): Closing peer: 92255276f9b144f3b14cc51ff4f450de I0517 06:44:36.262205 9073 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664): Closing peer: fc2430d5d05d4863b227c5cefc7ac460 I0517 06:44:36.262572 9073 consensus_queue.cc:204] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 0, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:36.259910 7976 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab [term 3 LEARNER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 580b6d55956e4c18bf9297327263345b (127.4.96.2) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:36.282994 8101 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b [term 3 FOLLOWER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0517 06:44:36.284255 8598 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0517 06:44:36.286020 8845 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de [term 3 LEARNER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0517 06:44:36.287930 9073 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0517 06:44:36.288394 9089 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:44:36.289113 9073 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0517 06:44:36.290745 8845 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de [term 3 LEARNER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 580b6d55956e4c18bf9297327263345b (127.4.96.2) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: NON_VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: true } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:36.293586 8100 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 5c25814d71724ad89c2b2965035156ab (127.4.96.1) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:36.292755 9091 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 5c25814d71724ad89c2b2965035156ab (127.4.96.1) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:36.300880 8598 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 5c25814d71724ad89c2b2965035156ab (127.4.96.1) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:36.302675 7975 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab [term 3 LEARNER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0517 06:44:36.302197 8845 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de [term 3 LEARNER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 5c25814d71724ad89c2b2965035156ab (127.4.96.1) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:36.304177 9098 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0517 06:44:36.307116 7975 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 5c25814d71724ad89c2b2965035156ab (127.4.96.1) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:36.308884 9098 raft_consensus.cc:866] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: attempting to promote NON_VOTER 92255276f9b144f3b14cc51ff4f450de to VOTER I0517 06:44:36.309119 9073 raft_consensus.cc:866] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: attempting to promote NON_VOTER 92255276f9b144f3b14cc51ff4f450de to VOTER I0517 06:44:36.309823 9098 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 5c25814d71724ad89c2b2965035156ab (127.4.96.1:43055): Closing peer: 5c25814d71724ad89c2b2965035156ab I0517 06:44:36.309980 9098 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 580b6d55956e4c18bf9297327263345b (127.4.96.2:33256): Closing peer: 580b6d55956e4c18bf9297327263345b I0517 06:44:36.310088 9098 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer 92255276f9b144f3b14cc51ff4f450de (127.4.96.8:55767): Closing peer: 92255276f9b144f3b14cc51ff4f450de I0517 06:44:36.310189 9098 consensus_peers.cc:456] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 -> Peer fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6:47664): Closing peer: fc2430d5d05d4863b227c5cefc7ac460 I0517 06:44:36.310592 9098 consensus_queue.cc:204] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 0, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: false } } unsafe_config_change: true W0517 06:44:36.313891 9073 raft_consensus.cc:870] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2: Unable to promote non-voter 92255276f9b144f3b14cc51ff4f450de: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true. Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: false } } unsafe_config_change: true I0517 06:44:36.315423 7976 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab [term 3 FOLLOWER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch) I0517 06:44:36.315779 8100 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b [term 3 FOLLOWER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch) I0517 06:44:36.316449 9089 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0517 06:44:36.316788 8598 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch) I0517 06:44:36.316881 9088 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0517 06:44:36.317853 9089 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0517 06:44:36.318262 8845 raft_consensus.cc:1057] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de [term 3 LEARNER]: Refusing update from remote peer dafb8d2b9a584a74b41c05787477c5c2: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch) I0517 06:44:36.320798 9091 consensus_queue.cc:963] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0517 06:44:36.323498 9089 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, 92255276f9b144f3b14cc51ff4f450de (127.4.96.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:36.325943 8982 catalog_manager.cc:3686] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de reported cstate change: config changed from index 7 to 8, 5c25814d71724ad89c2b2965035156ab (127.4.96.1) changed from NON_VOTER to VOTER, now has a pending config: VOTER dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5), VOTER fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6), VOTER 580b6d55956e4c18bf9297327263345b (127.4.96.2), VOTER 5c25814d71724ad89c2b2965035156ab (127.4.96.1), VOTER 92255276f9b144f3b14cc51ff4f450de (127.4.96.8). New cstate: current_term: 3 leader_uuid: "dafb8d2b9a584a74b41c05787477c5c2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: NON_VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: true } } unsafe_config_change: true } pending_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:36.329903 8845 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P 92255276f9b144f3b14cc51ff4f450de [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, 92255276f9b144f3b14cc51ff4f450de (127.4.96.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:36.329946 8100 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P 580b6d55956e4c18bf9297327263345b [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, 92255276f9b144f3b14cc51ff4f450de (127.4.96.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:36.333587 7975 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P 5c25814d71724ad89c2b2965035156ab [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, 92255276f9b144f3b14cc51ff4f450de (127.4.96.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:36.344318 8986 catalog_manager.cc:3686] T b6bea5c6c58641f8ab69083b20103628 P dafb8d2b9a584a74b41c05787477c5c2 reported cstate change: config changed from index 8 to 9, 92255276f9b144f3b14cc51ff4f450de (127.4.96.8) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER dafb8d2b9a584a74b41c05787477c5c2 (127.4.96.5), VOTER fc2430d5d05d4863b227c5cefc7ac460 (127.4.96.6), VOTER 580b6d55956e4c18bf9297327263345b (127.4.96.2), VOTER 5c25814d71724ad89c2b2965035156ab (127.4.96.1), VOTER 92255276f9b144f3b14cc51ff4f450de (127.4.96.8). New cstate: current_term: 3 leader_uuid: "dafb8d2b9a584a74b41c05787477c5c2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:44:36.346827 8598 raft_consensus.cc:2658] T b6bea5c6c58641f8ab69083b20103628 P fc2430d5d05d4863b227c5cefc7ac460 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, 92255276f9b144f3b14cc51ff4f450de (127.4.96.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 54438 } } peers { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 47664 } } peers { permanent_uuid: "580b6d55956e4c18bf9297327263345b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33256 } attrs { promote: false } } peers { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 43055 } attrs { promote: false } } peers { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 55767 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:36.387903 4480 kudu-admin-test.cc:704] Waiting for Master to see new config... I0517 06:44:36.399528 4480 kudu-admin-test.cc:709] Tablet locations: tablet_id: "b6bea5c6c58641f8ab69083b20103628" replicas { ts_info { permanent_uuid: "dafb8d2b9a584a74b41c05787477c5c2" rpc_addresses { host: "127.4.96.5" port: 54438 } } role: LEADER } replicas { ts_info { permanent_uuid: "fc2430d5d05d4863b227c5cefc7ac460" rpc_addresses { host: "127.4.96.6" port: 47664 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "580b6d55956e4c18bf9297327263345b" rpc_addresses { host: "127.4.96.2" port: 33256 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "5c25814d71724ad89c2b2965035156ab" rpc_addresses { host: "127.4.96.1" port: 43055 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "92255276f9b144f3b14cc51ff4f450de" rpc_addresses { host: "127.4.96.8" port: 55767 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0517 06:44:36.401085 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 7888 W0517 06:44:36.421861 8969 connection.cc:512] server connection from 127.4.96.1:41607 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:41607): Connection reset by peer (error 104) W0517 06:44:36.421917 7825 connection.cc:512] client connection to 127.4.96.1:43055 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:43055): Connection reset by peer (error 104) W0517 06:44:36.422969 7821 connection.cc:512] client connection to 127.4.96.1:43055 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:43055): Connection reset by peer (error 104) W0517 06:44:36.423746 8404 connection.cc:512] client connection to 127.4.96.1:43055 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:43055): Connection reset by peer (error 104) W0517 06:44:36.423763 8408 connection.cc:512] server connection from 127.4.96.1:36759 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:36759): Connection reset by peer (error 104) I0517 06:44:36.423892 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 8023 W0517 06:44:36.450989 7825 connection.cc:512] client connection to 127.4.96.2:33256 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:33256): Connection reset by peer (error 104) W0517 06:44:36.451047 8967 connection.cc:512] server connection from 127.4.96.2:56709 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:56709): Connection reset by peer (error 104) W0517 06:44:36.451802 8405 connection.cc:512] client connection to 127.4.96.2:33256 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:33256): Connection reset by peer (error 104) W0517 06:44:36.452615 8408 connection.cc:512] server connection from 127.4.96.2:41345 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:41345): Connection reset by peer (error 104) I0517 06:44:36.452842 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 8395 W0517 06:44:36.474727 8966 connection.cc:512] client connection to 127.4.96.5:54438 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:54438): Connection reset by peer (error 104) W0517 06:44:36.474757 8777 connection.cc:512] client connection to 127.4.96.5:54438 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:54438): Connection reset by peer (error 104) W0517 06:44:36.475605 8965 connection.cc:512] server connection from 127.4.96.5:43556 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:43556): Connection reset by peer (error 104) W0517 06:44:36.476263 8777 connection.cc:512] server connection from 127.4.96.5:53486 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:53486): Connection reset by peer (error 104) W0517 06:44:36.476263 8529 connection.cc:512] server connection from 127.4.96.5:54595 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:54595): Connection reset by peer (error 104) W0517 06:44:36.476955 8529 connection.cc:512] client connection to 127.4.96.5:54438 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:54438): Connection reset by peer (error 104) W0517 06:44:36.477107 7825 connection.cc:512] client connection to 127.4.96.5:54438 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:54438): Connection reset by peer (error 104) I0517 06:44:36.478139 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 8520 W0517 06:44:36.497500 8965 connection.cc:512] server connection from 127.4.96.6:45411 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.6:45411): Connection reset by peer (error 104) W0517 06:44:36.497485 7825 connection.cc:512] client connection to 127.4.96.6:47664 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.6:47664): Connection reset by peer (error 104) I0517 06:44:36.498540 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 8768 W0517 06:44:36.498553 7824 connection.cc:512] client connection to 127.4.96.6:47664 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.6:47664): Connection reset by peer (error 104) W0517 06:44:36.521407 8966 connection.cc:512] server connection from 127.4.96.8:55213 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.8:55213): Connection reset by peer (error 104) W0517 06:44:36.521512 7825 connection.cc:512] client connection to 127.4.96.8:55767 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.8:55767): Connection reset by peer (error 104) I0517 06:44:36.522378 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 8957 W0517 06:44:36.553153 7825 connection.cc:512] client connection to 127.4.96.62:54038 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:54038): Connection reset by peer (error 104) [ OK ] AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig (10163 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig I0517 06:44:36.573807 4480 test_util.cc:200] Using random seed: 725392979 I0517 06:44:36.576691 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:44:36.576786 4480 ts_itest-base.cc:114] -------------- I0517 06:44:36.576865 4480 ts_itest-base.cc:115] 5 tablet servers I0517 06:44:36.576939 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:44:36.577013 4480 ts_itest-base.cc:117] -------------- I0517 06:44:36.579227 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:40202 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:36.691586 9112 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:36.691891 9112 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:36.700846 9112 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:44:36.700965 9112 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:36.701061 9112 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:44:36.701172 9112 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:44:36.715085 9112 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:40202 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:36.722782 9112 master_main.cc:80] Initializing master server... I0517 06:44:36.724287 9112 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 137319us I0517 06:44:36.725011 9112 fs_manager.cc:260] Metadata directory not provided I0517 06:44:36.725133 9112 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:36.725327 9112 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:44:36.725435 9112 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:36.732681 9112 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "952ce954d44a4751b333513bc1e723ec" format_stamp: "Formatted at 2018-05-17 06:44:36 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:36.733322 9112 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "952ce954d44a4751b333513bc1e723ec" format_stamp: "Formatted at 2018-05-17 06:44:36 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:36.743242 9112 fs_manager.cc:495] Time spent creating directory manager: real 0.010s user 0.004s sys 0.008s I0517 06:44:36.744344 9112 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:36.744735 9112 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:36.751091 9112 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:44:36.751195 9112 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "952ce954d44a4751b333513bc1e723ec" format_stamp: "Formatted at 2018-05-17 06:44:36 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:36.751407 9112 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:36.959614 9112 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:36.961293 9112 master_main.cc:83] Starting Master server... I0517 06:44:37.000694 9112 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:40202 I0517 06:44:37.001526 9112 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:44:37.001623 9112 webserver.cc:184] Document root disabled I0517 06:44:37.002895 9112 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:43421/ I0517 06:44:37.003604 9112 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:44:37.006134 9173 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:37.011731 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 9112 I0517 06:44:37.012645 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:40202 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:37.019670 9173 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: Bootstrap starting. I0517 06:44:37.021873 9173 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: No blocks or log segments found. Creating new log. I0517 06:44:37.023049 9173 log.cc:520] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: Log is configured to *not* fsync() on all Append() calls I0517 06:44:37.025370 9173 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: No bootstrap required, opened a new log I0517 06:44:37.036890 9173 raft_consensus.cc:303] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } I0517 06:44:37.037202 9173 raft_consensus.cc:329] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:37.037355 9173 raft_consensus.cc:604] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952ce954d44a4751b333513bc1e723ec, State: Initialized, Role: FOLLOWER I0517 06:44:37.037811 9173 consensus_queue.cc:227] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } I0517 06:44:37.038182 9173 raft_consensus.cc:343] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:44:37.038354 9173 raft_consensus.cc:435] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:44:37.038513 9173 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: Snoozing failure detection for 2.166s (starting election) I0517 06:44:37.038667 9173 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:37.041028 9173 raft_consensus.cc:457] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } I0517 06:44:37.041466 9173 leader_election.cc:270] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:44:37.043162 9176 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: Snoozing failure detection for 2.144s (election complete) I0517 06:44:37.043431 9176 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:44:37.055124 9176 raft_consensus.cc:571] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 1 LEADER]: Becoming Leader. State: Replica: 952ce954d44a4751b333513bc1e723ec, State: Running, Role: LEADER I0517 06:44:37.055536 9176 consensus_queue.cc:204] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } I0517 06:44:37.058714 9173 sys_catalog.cc:335] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "952ce954d44a4751b333513bc1e723ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } } I0517 06:44:37.059134 9173 sys_catalog.cc:338] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: This master's current role is: LEADER WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:37.123211 9174 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:37.123555 9174 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:37.131482 9174 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:37.131814 9174 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:44:37.059924 9177 sys_catalog.cc:335] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "952ce954d44a4751b333513bc1e723ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } } I0517 06:44:37.144918 9177 sys_catalog.cc:338] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: This master's current role is: LEADER I0517 06:44:37.064004 9178 sys_catalog.cc:335] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 952ce954d44a4751b333513bc1e723ec. Latest consensus state: current_term: 1 leader_uuid: "952ce954d44a4751b333513bc1e723ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } } I0517 06:44:37.148095 9178 sys_catalog.cc:338] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: This master's current role is: LEADER I0517 06:44:37.148339 9174 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:40202 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:37.149637 9173 sys_catalog.cc:422] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: configured and running, proceeding with master startup. I0517 06:44:37.149816 9180 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:44:37.151477 9112 master_main.cc:86] Master server successfully started. I0517 06:44:37.157187 9174 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:37.160001 9180 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:44:37.162051 9174 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 137819us I0517 06:44:37.162504 9174 fs_manager.cc:260] Metadata directory not provided I0517 06:44:37.162637 9174 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:37.162842 9174 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:44:37.162963 9174 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:37.170667 9174 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "120b69a6c7e24e98ad83d42c78805f4e" format_stamp: "Formatted at 2018-05-17 06:44:37 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:37.171340 9174 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "120b69a6c7e24e98ad83d42c78805f4e" format_stamp: "Formatted at 2018-05-17 06:44:37 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:37.193506 9174 fs_manager.cc:495] Time spent creating directory manager: real 0.022s user 0.008s sys 0.012s I0517 06:44:37.194694 9174 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:37.195098 9174 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:37.199579 9180 catalog_manager.cc:899] Generated new certificate authority record I0517 06:44:37.201823 9180 catalog_manager.cc:1021] Loading token signing keys... I0517 06:44:37.213301 9174 fs_manager.cc:417] Time spent opening block manager: real 0.011s user 0.004s sys 0.008s I0517 06:44:37.213482 9174 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "120b69a6c7e24e98ad83d42c78805f4e" format_stamp: "Formatted at 2018-05-17 06:44:37 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:37.213730 9174 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:37.214360 9180 catalog_manager.cc:3962] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: Generated new TSK 0 I0517 06:44:37.449046 9174 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:37.451727 9174 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:37.452816 9174 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:37.555498 9174 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:41524 I0517 06:44:37.556401 9174 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:44:37.556504 9174 webserver.cc:184] Document root disabled I0517 06:44:37.557883 9174 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:58273/ I0517 06:44:37.558655 9174 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:44:37.566788 9174 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:37.567477 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 9174 I0517 06:44:37.568734 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:40202 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:37.592658 9305 heartbeater.cc:346] Connected to a master server at 127.4.96.62:40202 I0517 06:44:37.592952 9305 heartbeater.cc:426] Registering TS with master... I0517 06:44:37.593503 9305 heartbeater.cc:475] Master 127.4.96.62:40202 requested a full tablet report, sending... I0517 06:44:37.595428 9140 ts_manager.cc:81] Registered new tserver with Master: 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1:41524) I0517 06:44:37.597883 9140 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:34097 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:37.681157 9309 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:37.681511 9309 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:37.689520 9309 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:37.689846 9309 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:44:37.702641 9309 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:40202 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:37.710678 9309 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:37.712466 9309 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 137819us I0517 06:44:37.712888 9309 fs_manager.cc:260] Metadata directory not provided I0517 06:44:37.713011 9309 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:37.713232 9309 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:44:37.713423 9309 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:37.721009 9309 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "44dc106a097740c3b1e984dd2ba3d664" format_stamp: "Formatted at 2018-05-17 06:44:37 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:37.721680 9309 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "44dc106a097740c3b1e984dd2ba3d664" format_stamp: "Formatted at 2018-05-17 06:44:37 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:37.731201 9309 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:44:37.732298 9309 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:37.732683 9309 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:37.739259 9309 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.004s sys 0.004s I0517 06:44:37.739405 9309 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "44dc106a097740c3b1e984dd2ba3d664" format_stamp: "Formatted at 2018-05-17 06:44:37 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:37.739614 9309 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:37.928050 9309 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:37.931298 9309 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:37.932610 9309 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:38.036486 9309 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:42545 I0517 06:44:38.037364 9309 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:44:38.037467 9309 webserver.cc:184] Document root disabled I0517 06:44:38.038842 9309 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:47129/ I0517 06:44:38.039515 9309 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:44:38.042377 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 9309 I0517 06:44:38.043633 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:40202 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:38.044394 9309 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:38.060411 9430 heartbeater.cc:346] Connected to a master server at 127.4.96.62:40202 I0517 06:44:38.060659 9430 heartbeater.cc:426] Registering TS with master... I0517 06:44:38.061115 9430 heartbeater.cc:475] Master 127.4.96.62:40202 requested a full tablet report, sending... I0517 06:44:38.062470 9140 ts_manager.cc:81] Registered new tserver with Master: 44dc106a097740c3b1e984dd2ba3d664 (127.4.96.2:42545) I0517 06:44:38.063565 9140 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:36055 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:38.201792 9433 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:38.202278 9433 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:38.216506 9433 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:38.216990 9433 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:44:38.237380 9433 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:40202 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:38.250500 9433 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:38.252909 9433 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 138319us I0517 06:44:38.253526 9433 fs_manager.cc:260] Metadata directory not provided I0517 06:44:38.253690 9433 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:38.253978 9433 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:44:38.254148 9433 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:38.266887 9433 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "101a15d6048446abbf30c04b2944d96d" format_stamp: "Formatted at 2018-05-17 06:44:38 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:38.267819 9433 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "101a15d6048446abbf30c04b2944d96d" format_stamp: "Formatted at 2018-05-17 06:44:38 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:38.281762 9433 fs_manager.cc:495] Time spent creating directory manager: real 0.014s user 0.012s sys 0.004s I0517 06:44:38.283416 9433 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:38.283916 9433 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:38.294301 9433 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:44:38.294446 9433 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "101a15d6048446abbf30c04b2944d96d" format_stamp: "Formatted at 2018-05-17 06:44:38 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:38.294718 9433 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:38.499243 9433 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:38.502349 9433 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:38.503504 9433 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:38.600914 9305 heartbeater.cc:467] Master 127.4.96.62:40202 was elected leader, sending a full tablet report... I0517 06:44:38.613688 9433 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:39331 I0517 06:44:38.614598 9433 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:44:38.614703 9433 webserver.cc:184] Document root disabled I0517 06:44:38.615893 9433 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:44390/ I0517 06:44:38.616581 9433 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:44:38.619040 9433 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:38.623541 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 9433 I0517 06:44:38.624836 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:40202 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:38.638604 9554 heartbeater.cc:346] Connected to a master server at 127.4.96.62:40202 I0517 06:44:38.638842 9554 heartbeater.cc:426] Registering TS with master... I0517 06:44:38.639277 9554 heartbeater.cc:475] Master 127.4.96.62:40202 requested a full tablet report, sending... I0517 06:44:38.640607 9140 ts_manager.cc:81] Registered new tserver with Master: 101a15d6048446abbf30c04b2944d96d (127.4.96.3:39331) I0517 06:44:38.641681 9140 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:36247 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:38.728101 9558 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:38.728412 9558 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:38.736685 9558 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:38.736990 9558 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:44:38.749341 9558 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:40202 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:38.757321 9558 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:38.759065 9558 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 138319us I0517 06:44:38.759462 9558 fs_manager.cc:260] Metadata directory not provided I0517 06:44:38.759567 9558 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:38.759760 9558 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:44:38.759882 9558 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:38.767315 9558 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "4d50d25c58304e38b0d506069d0002ff" format_stamp: "Formatted at 2018-05-17 06:44:38 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:38.767925 9558 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "4d50d25c58304e38b0d506069d0002ff" format_stamp: "Formatted at 2018-05-17 06:44:38 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:38.776901 9558 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:44:38.778041 9558 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:38.778400 9558 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:38.784610 9558 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:44:38.784718 9558 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "4d50d25c58304e38b0d506069d0002ff" format_stamp: "Formatted at 2018-05-17 06:44:38 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:38.784896 9558 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:38.942536 9558 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:38.944856 9558 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:38.945763 9558 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:39.033478 9558 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:56258 I0517 06:44:39.034375 9558 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:44:39.034484 9558 webserver.cc:184] Document root disabled I0517 06:44:39.035662 9558 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:42596/ I0517 06:44:39.036340 9558 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:44:39.038923 9558 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:39.041056 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 9558 I0517 06:44:39.042063 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.4.96.5:0 --local_ip_for_outbound_sockets=127.4.96.5 --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:40202 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:39.056185 9679 heartbeater.cc:346] Connected to a master server at 127.4.96.62:40202 I0517 06:44:39.056435 9679 heartbeater.cc:426] Registering TS with master... I0517 06:44:39.056852 9679 heartbeater.cc:475] Master 127.4.96.62:40202 requested a full tablet report, sending... I0517 06:44:39.058151 9140 ts_manager.cc:81] Registered new tserver with Master: 4d50d25c58304e38b0d506069d0002ff (127.4.96.4:56258) I0517 06:44:39.059222 9140 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:55778 I0517 06:44:39.065979 9430 heartbeater.cc:467] Master 127.4.96.62:40202 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:39.146092 9682 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:39.146421 9682 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:39.154352 9682 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:39.154656 9682 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.5 I0517 06:44:39.167176 9682 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:40202 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.5 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:39.175048 9682 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:39.176772 9682 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 138819us I0517 06:44:39.177176 9682 fs_manager.cc:260] Metadata directory not provided I0517 06:44:39.177284 9682 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:39.177503 9682 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0517 06:44:39.177614 9682 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:39.184993 9682 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" format_stamp: "Formatted at 2018-05-17 06:44:39 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:39.185606 9682 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" format_stamp: "Formatted at 2018-05-17 06:44:39 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:39.194808 9682 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:44:39.195909 9682 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:39.196303 9682 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:39.202589 9682 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:44:39.202695 9682 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" format_stamp: "Formatted at 2018-05-17 06:44:39 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:39.202879 9682 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:39.357890 9682 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:39.360227 9682 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:39.361148 9682 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:39.445909 9682 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.5:33454 I0517 06:44:39.446799 9682 webserver.cc:173] Starting webserver on 127.4.96.5:0 I0517 06:44:39.446892 9682 webserver.cc:184] Document root disabled I0517 06:44:39.448297 9682 webserver.cc:311] Webserver started. Bound to: http://127.4.96.5:60725/ I0517 06:44:39.448992 9682 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb I0517 06:44:39.452219 9682 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:39.456804 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 9682 I0517 06:44:39.470252 9803 heartbeater.cc:346] Connected to a master server at 127.4.96.62:40202 I0517 06:44:39.470487 9803 heartbeater.cc:426] Registering TS with master... I0517 06:44:39.470921 9803 heartbeater.cc:475] Master 127.4.96.62:40202 requested a full tablet report, sending... I0517 06:44:39.472174 9140 ts_manager.cc:81] Registered new tserver with Master: e59d0eda53f7443bb8b30c8f84fa5a37 (127.4.96.5:33454) I0517 06:44:39.473215 9140 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.5:54942 I0517 06:44:39.475035 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:44:39.491722 9140 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37315: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:44:39.533938 9616 tablet_service.cc:759] Processing CreateTablet for tablet 04d150767ee548d6b6d776284666ac51 (table=TestTable [id=6eafaf0c70a344ef85e470ded1fd4cab]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:39.533836 9242 tablet_service.cc:759] Processing CreateTablet for tablet 04d150767ee548d6b6d776284666ac51 (table=TestTable [id=6eafaf0c70a344ef85e470ded1fd4cab]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:39.534838 9616 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 04d150767ee548d6b6d776284666ac51. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:39.535140 9242 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 04d150767ee548d6b6d776284666ac51. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:39.541968 9367 tablet_service.cc:759] Processing CreateTablet for tablet 04d150767ee548d6b6d776284666ac51 (table=TestTable [id=6eafaf0c70a344ef85e470ded1fd4cab]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:39.542845 9367 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 04d150767ee548d6b6d776284666ac51. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:39.548033 9242 ts_tablet_manager.cc:1173] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:39.550269 9616 ts_tablet_manager.cc:1173] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:39.554376 9824 ts_tablet_manager.cc:1028] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff: Bootstrapping tablet I0517 06:44:39.554752 9824 tablet_bootstrap.cc:436] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff: Bootstrap starting. I0517 06:44:39.556813 9823 ts_tablet_manager.cc:1028] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e: Bootstrapping tablet I0517 06:44:39.556932 9824 tablet_bootstrap.cc:587] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff: No blocks or log segments found. Creating new log. I0517 06:44:39.557273 9823 tablet_bootstrap.cc:436] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e: Bootstrap starting. I0517 06:44:39.558387 9824 log.cc:520] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff: Log is configured to *not* fsync() on all Append() calls I0517 06:44:39.558490 9367 ts_tablet_manager.cc:1173] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:39.560508 9823 tablet_bootstrap.cc:587] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e: No blocks or log segments found. Creating new log. I0517 06:44:39.562021 9823 log.cc:520] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e: Log is configured to *not* fsync() on all Append() calls I0517 06:44:39.565304 9826 ts_tablet_manager.cc:1028] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Bootstrapping tablet I0517 06:44:39.565971 9823 tablet_bootstrap.cc:436] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e: No bootstrap required, opened a new log I0517 06:44:39.566175 9824 tablet_bootstrap.cc:436] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff: No bootstrap required, opened a new log I0517 06:44:39.566244 9823 ts_tablet_manager.cc:1045] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s I0517 06:44:39.566519 9824 ts_tablet_manager.cc:1045] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s I0517 06:44:39.566978 9826 tablet_bootstrap.cc:436] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Bootstrap starting. I0517 06:44:39.569950 9826 tablet_bootstrap.cc:587] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: No blocks or log segments found. Creating new log. I0517 06:44:39.571523 9826 log.cc:520] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Log is configured to *not* fsync() on all Append() calls I0517 06:44:39.574326 9826 tablet_bootstrap.cc:436] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: No bootstrap required, opened a new log I0517 06:44:39.574698 9826 ts_tablet_manager.cc:1045] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s I0517 06:44:39.577463 9823 raft_consensus.cc:303] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d50d25c58304e38b0d506069d0002ff" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56258 } } peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } } I0517 06:44:39.578300 9823 raft_consensus.cc:329] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:39.578513 9823 raft_consensus.cc:604] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120b69a6c7e24e98ad83d42c78805f4e, State: Initialized, Role: FOLLOWER I0517 06:44:39.579171 9823 consensus_queue.cc:227] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d50d25c58304e38b0d506069d0002ff" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56258 } } peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } } I0517 06:44:39.581606 9824 raft_consensus.cc:303] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d50d25c58304e38b0d506069d0002ff" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56258 } } peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } } I0517 06:44:39.582892 9823 ts_tablet_manager.cc:1073] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s I0517 06:44:39.583477 9824 raft_consensus.cc:329] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:39.583624 9824 raft_consensus.cc:604] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d50d25c58304e38b0d506069d0002ff, State: Initialized, Role: FOLLOWER I0517 06:44:39.584029 9824 consensus_queue.cc:227] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d50d25c58304e38b0d506069d0002ff" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56258 } } peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } } I0517 06:44:39.586299 9679 heartbeater.cc:467] Master 127.4.96.62:40202 was elected leader, sending a full tablet report... I0517 06:44:39.587785 9824 ts_tablet_manager.cc:1073] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s I0517 06:44:39.587764 9826 raft_consensus.cc:303] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d50d25c58304e38b0d506069d0002ff" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56258 } } peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } } I0517 06:44:39.588492 9826 raft_consensus.cc:329] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:39.588709 9826 raft_consensus.cc:604] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44dc106a097740c3b1e984dd2ba3d664, State: Initialized, Role: FOLLOWER I0517 06:44:39.589164 9826 consensus_queue.cc:227] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d50d25c58304e38b0d506069d0002ff" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56258 } } peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } } I0517 06:44:39.591264 9826 ts_tablet_manager.cc:1073] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s I0517 06:44:39.643901 9554 heartbeater.cc:467] Master 127.4.96.62:40202 was elected leader, sending a full tablet report... I0517 06:44:39.679739 9831 raft_consensus.cc:435] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:44:39.679939 9831 raft_consensus.cc:2704] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Snoozing failure detection for 2.215s (starting election) I0517 06:44:39.680126 9831 raft_consensus.cc:457] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d50d25c58304e38b0d506069d0002ff" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56258 } } peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } } I0517 06:44:39.680971 9831 leader_election.cc:243] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 4d50d25c58304e38b0d506069d0002ff (127.4.96.4:56258) I0517 06:44:39.681370 9831 leader_election.cc:243] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1:41524) I0517 06:44:39.691890 9636 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "04d150767ee548d6b6d776284666ac51" candidate_uuid: "44dc106a097740c3b1e984dd2ba3d664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d50d25c58304e38b0d506069d0002ff" is_pre_election: true I0517 06:44:39.692095 9262 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "04d150767ee548d6b6d776284666ac51" candidate_uuid: "44dc106a097740c3b1e984dd2ba3d664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120b69a6c7e24e98ad83d42c78805f4e" is_pre_election: true I0517 06:44:39.692306 9636 raft_consensus.cc:2704] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff: Snoozing failure detection for 2.171s (vote granted) I0517 06:44:39.692502 9636 raft_consensus.cc:2219] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44dc106a097740c3b1e984dd2ba3d664 in term 0. I0517 06:44:39.692548 9262 raft_consensus.cc:2704] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e: Snoozing failure detection for 1.999s (vote granted) I0517 06:44:39.692739 9262 raft_consensus.cc:2219] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44dc106a097740c3b1e984dd2ba3d664 in term 0. I0517 06:44:39.693343 9321 leader_election.cc:379] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 4d50d25c58304e38b0d506069d0002ff (127.4.96.4:56258) I0517 06:44:39.693523 9321 leader_election.cc:270] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:44:39.693753 9319 leader_election.cc:379] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1:41524) I0517 06:44:39.693917 9831 raft_consensus.cc:2704] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Snoozing failure detection for 2.086s (election complete) I0517 06:44:39.746557 9831 raft_consensus.cc:2511] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:44:39.746695 9831 raft_consensus.cc:435] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:44:39.746831 9831 raft_consensus.cc:2704] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Snoozing failure detection for 1.894s (starting election) I0517 06:44:39.746953 9831 raft_consensus.cc:2756] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:39.749784 9831 raft_consensus.cc:457] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d50d25c58304e38b0d506069d0002ff" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56258 } } peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } } I0517 06:44:39.750458 9831 leader_election.cc:243] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [CANDIDATE]: Term 1 election: Requesting vote from peer 4d50d25c58304e38b0d506069d0002ff (127.4.96.4:56258) I0517 06:44:39.750751 9831 leader_election.cc:243] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [CANDIDATE]: Term 1 election: Requesting vote from peer 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1:41524) I0517 06:44:39.751487 9636 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "04d150767ee548d6b6d776284666ac51" candidate_uuid: "44dc106a097740c3b1e984dd2ba3d664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d50d25c58304e38b0d506069d0002ff" I0517 06:44:39.751809 9636 raft_consensus.cc:2756] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:39.751705 9262 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "04d150767ee548d6b6d776284666ac51" candidate_uuid: "44dc106a097740c3b1e984dd2ba3d664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120b69a6c7e24e98ad83d42c78805f4e" I0517 06:44:39.752002 9636 raft_consensus.cc:2704] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff: Snoozing failure detection for 1.839s (vote granted) I0517 06:44:39.752024 9262 raft_consensus.cc:2756] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:39.752221 9262 raft_consensus.cc:2704] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e: Snoozing failure detection for 2.005s (vote granted) I0517 06:44:39.754720 9636 raft_consensus.cc:2219] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44dc106a097740c3b1e984dd2ba3d664 in term 1. I0517 06:44:39.754992 9262 raft_consensus.cc:2219] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44dc106a097740c3b1e984dd2ba3d664 in term 1. I0517 06:44:39.755439 9321 leader_election.cc:379] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [CANDIDATE]: Term 1 election: Vote granted by peer 4d50d25c58304e38b0d506069d0002ff (127.4.96.4:56258) I0517 06:44:39.755599 9321 leader_election.cc:270] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:44:39.755777 9319 leader_election.cc:379] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [CANDIDATE]: Term 1 election: Vote granted by peer 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1:41524) I0517 06:44:39.756012 9831 raft_consensus.cc:2704] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Snoozing failure detection for 2.129s (election complete) I0517 06:44:39.756223 9831 raft_consensus.cc:2511] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:44:39.757386 9831 raft_consensus.cc:571] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 1 LEADER]: Becoming Leader. State: Replica: 44dc106a097740c3b1e984dd2ba3d664, State: Running, Role: LEADER I0517 06:44:39.757755 9831 consensus_queue.cc:204] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d50d25c58304e38b0d506069d0002ff" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56258 } } peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } } I0517 06:44:39.762609 9139 catalog_manager.cc:3686] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 reported cstate change: term changed from 0 to 1, leader changed from to 44dc106a097740c3b1e984dd2ba3d664 (127.4.96.2). New cstate: current_term: 1 leader_uuid: "44dc106a097740c3b1e984dd2ba3d664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d50d25c58304e38b0d506069d0002ff" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56258 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } health_report { overall_health: UNKNOWN } } } W0517 06:44:39.792851 9431 tablet.cc:1678] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:39.818778 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:44:39.821382 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 120b69a6c7e24e98ad83d42c78805f4e to finish bootstrapping I0517 06:44:39.835400 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 44dc106a097740c3b1e984dd2ba3d664 to finish bootstrapping I0517 06:44:39.848212 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 101a15d6048446abbf30c04b2944d96d to finish bootstrapping I0517 06:44:39.864362 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4d50d25c58304e38b0d506069d0002ff to finish bootstrapping W0517 06:44:39.789172 9680 tablet.cc:1678] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:39.876260 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e59d0eda53f7443bb8b30c8f84fa5a37 to finish bootstrapping I0517 06:44:39.887619 4480 kudu-admin-test.cc:740] Waiting for Master to see the current replicas... I0517 06:44:39.890202 4480 kudu-admin-test.cc:747] Tablet locations: tablet_id: "04d150767ee548d6b6d776284666ac51" replicas { ts_info { permanent_uuid: "4d50d25c58304e38b0d506069d0002ff" rpc_addresses { host: "127.4.96.4" port: 56258 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" rpc_addresses { host: "127.4.96.2" port: 42545 } } role: LEADER } replicas { ts_info { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" rpc_addresses { host: "127.4.96.1" port: 41524 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } W0517 06:44:39.817602 9306 tablet.cc:1678] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:40.192514 9636 raft_consensus.cc:1057] T 04d150767ee548d6b6d776284666ac51 P 4d50d25c58304e38b0d506069d0002ff [term 1 FOLLOWER]: Refusing update from remote peer 44dc106a097740c3b1e984dd2ba3d664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:40.193610 9831 consensus_queue.cc:963] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d50d25c58304e38b0d506069d0002ff" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56258 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:40.202553 9262 raft_consensus.cc:1057] T 04d150767ee548d6b6d776284666ac51 P 120b69a6c7e24e98ad83d42c78805f4e [term 1 FOLLOWER]: Refusing update from remote peer 44dc106a097740c3b1e984dd2ba3d664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:40.203693 9831 consensus_queue.cc:963] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:40.213510 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 9174 W0517 06:44:40.232964 9122 connection.cc:512] client connection to 127.4.96.1:41524 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:41524): Connection reset by peer (error 104) W0517 06:44:40.233077 9319 connection.cc:512] client connection to 127.4.96.1:41524 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:41524): Connection reset by peer (error 104) W0517 06:44:40.233911 9124 connection.cc:512] server connection from 127.4.96.1:34097 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:34097): Connection reset by peer (error 104) W0517 06:44:40.234071 9111 connection.cc:512] client connection to 127.4.96.1:41524 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:41524): Connection reset by peer (error 104) W0517 06:44:40.234886 9109 connection.cc:512] client connection to 127.4.96.1:41524 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:41524): Connection reset by peer (error 104) I0517 06:44:40.235039 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 9558 W0517 06:44:40.252656 9124 connection.cc:512] server connection from 127.4.96.4:55778 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:55778): Connection reset by peer (error 104) W0517 06:44:40.252660 9321 connection.cc:512] client connection to 127.4.96.4:56258 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:56258): Connection reset by peer (error 104) W0517 06:44:40.253196 9124 connection.cc:512] client connection to 127.4.96.4:56258 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:56258): Connection reset by peer (error 104) W0517 06:44:40.253502 9111 connection.cc:512] client connection to 127.4.96.4:56258 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:56258): Connection reset by peer (error 104) I0517 06:44:40.255980 9387 consensus_peers.cc:456] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1:41524): Closing peer: 120b69a6c7e24e98ad83d42c78805f4e I0517 06:44:40.256152 9387 consensus_peers.cc:456] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer 4d50d25c58304e38b0d506069d0002ff (127.4.96.4:56258): Closing peer: 4d50d25c58304e38b0d506069d0002ff I0517 06:44:40.256567 9387 consensus_queue.cc:204] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } } W0517 06:44:40.259807 9319 consensus_peers.cc:434] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1:41524): Couldn't send request to peer 120b69a6c7e24e98ad83d42c78805f4e for tablet 04d150767ee548d6b6d776284666ac51. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.1:41524: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:40.475420 9803 heartbeater.cc:467] Master 127.4.96.62:40202 was elected leader, sending a full tablet report... W0517 06:44:40.785521 9319 consensus_peers.cc:434] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1:41524): Couldn't send request to peer 120b69a6c7e24e98ad83d42c78805f4e for tablet 04d150767ee548d6b6d776284666ac51. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.1:41524: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0517 06:44:41.289795 9319 consensus_peers.cc:434] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1:41524): Couldn't send request to peer 120b69a6c7e24e98ad83d42c78805f4e for tablet 04d150767ee548d6b6d776284666ac51. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.1:41524: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0517 06:44:41.907534 9319 consensus_peers.cc:434] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1:41524): Couldn't send request to peer 120b69a6c7e24e98ad83d42c78805f4e for tablet 04d150767ee548d6b6d776284666ac51. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.1:41524: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. I0517 06:44:42.256731 4480 kudu-admin-test.cc:769] Change Config Op timed out, Sending a Replace config command when change config op is pending on the leader. W0517 06:44:42.366468 9319 consensus_peers.cc:434] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1:41524): Couldn't send request to peer 120b69a6c7e24e98ad83d42c78805f4e for tablet 04d150767ee548d6b6d776284666ac51. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.1:41524: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0517 06:44:42.370960 9851 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:44:42.566342 9387 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "44dc106a097740c3b1e984dd2ba3d664" tablet_id: "04d150767ee548d6b6d776284666ac51" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" } } from {username='slave'} at 127.0.0.1:35681 W0517 06:44:42.566793 9387 raft_consensus.cc:1891] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 1 LEADER]: Replica has a pending config, but the new config will be unsafely changed anyway. Currently pending config on the node: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } } W0517 06:44:42.567347 9387 raft_consensus.cc:1990] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 1 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d50d25c58304e38b0d506069d0002ff" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56258 } } peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } unsafe_config_change: true I0517 06:44:42.567960 9387 raft_consensus.cc:2751] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 1 LEADER]: Stepping down as leader of term 1 I0517 06:44:42.568099 9387 raft_consensus.cc:604] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44dc106a097740c3b1e984dd2ba3d664, State: Running, Role: LEADER I0517 06:44:42.568415 9387 consensus_queue.cc:227] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } } I0517 06:44:42.568797 9387 consensus_peers.cc:456] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1:41524): Closing peer: 120b69a6c7e24e98ad83d42c78805f4e I0517 06:44:42.569020 9387 raft_consensus.cc:2756] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:44:42.572955 9387 raft_consensus.cc:2823] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 2 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } }; New pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } unsafe_config_change: true I0517 06:44:43.587604 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 9112 W0517 06:44:43.613167 9318 connection.cc:512] client connection to 127.4.96.62:40202 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:40202): Connection reset by peer (error 104) W0517 06:44:43.613270 9691 connection.cc:512] client connection to 127.4.96.62:40202 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:40202): Connection reset by peer (error 104) W0517 06:44:43.613270 9442 connection.cc:512] client connection to 127.4.96.62:40202 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:40202): Connection reset by peer (error 104) W0517 06:44:43.614408 9321 connection.cc:512] server connection from 127.0.0.1:35672 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35672): Connection reset by peer (error 104) W0517 06:44:43.614467 9111 connection.cc:512] client connection to 127.4.96.62:40202 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:40202): Connection reset by peer (error 104) W0517 06:44:43.615676 9809 connection.cc:512] client connection to 127.4.96.62:40202 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:40202): Connection reset by peer (error 104) I0517 06:44:43.616082 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:40202 --webserver_interface=127.4.96.62 --webserver_port=43421 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} W0517 06:44:43.653964 9554 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:40202: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.96.62:40202: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:43.729784 9867 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:43.730100 9867 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:43.738936 9867 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:44:43.739053 9867 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:43.739153 9867 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:44:43.739262 9867 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:44:43.753048 9867 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:40202 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=43421 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:43.760702 9867 master_main.cc:80] Initializing master server... I0517 06:44:43.762212 9867 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 140819us I0517 06:44:43.762928 9867 fs_manager.cc:260] Metadata directory not provided I0517 06:44:43.763064 9867 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:43.768385 9867 fs_manager.cc:397] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s I0517 06:44:43.769011 9867 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:43.769407 9867 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:43.775815 9867 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:44:43.775924 9867 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "952ce954d44a4751b333513bc1e723ec" format_stamp: "Formatted at 2018-05-17 06:44:36 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:43.777037 9867 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:43.959998 9867 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:43.961695 9867 master_main.cc:83] Starting Master server... I0517 06:44:44.002535 9867 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:40202 I0517 06:44:44.003381 9867 webserver.cc:173] Starting webserver on 127.4.96.62:43421 I0517 06:44:44.003477 9867 webserver.cc:184] Document root disabled I0517 06:44:44.004680 9867 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:43421/ I0517 06:44:44.005437 9867 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:44:44.009411 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 9867 I0517 06:44:44.015102 9929 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: Bootstrap starting. I0517 06:44:44.040702 9929 log.cc:520] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: Log is configured to *not* fsync() on all Append() calls I0517 06:44:44.052330 9929 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:44.052882 9929 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: Bootstrap complete. I0517 06:44:44.070530 9929 raft_consensus.cc:303] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } I0517 06:44:44.072098 9929 raft_consensus.cc:604] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952ce954d44a4751b333513bc1e723ec, State: Initialized, Role: FOLLOWER I0517 06:44:44.072585 9929 consensus_queue.cc:227] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } I0517 06:44:44.072963 9929 raft_consensus.cc:343] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:44:44.073107 9929 raft_consensus.cc:435] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:44:44.073256 9929 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: Snoozing failure detection for 1.947s (starting election) I0517 06:44:44.073400 9929 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 1 FOLLOWER]: Advancing to term 2 I0517 06:44:44.074329 9933 raft_consensus.cc:435] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0517 06:44:44.074602 9933 raft_consensus.cc:2704] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Snoozing failure detection for 1.770s (starting election) I0517 06:44:44.074805 9933 raft_consensus.cc:457] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } unsafe_config_change: true I0517 06:44:44.075248 9933 leader_election.cc:270] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0517 06:44:44.076184 9933 raft_consensus.cc:2704] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Snoozing failure detection for 1.701s (election complete) I0517 06:44:44.076370 9933 raft_consensus.cc:2511] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0517 06:44:44.076503 9933 raft_consensus.cc:435] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0517 06:44:44.076653 9933 raft_consensus.cc:2704] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Snoozing failure detection for 2.171s (starting election) I0517 06:44:44.076812 9933 raft_consensus.cc:2756] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 2 FOLLOWER]: Advancing to term 3 I0517 06:44:44.077682 9929 raft_consensus.cc:457] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } I0517 06:44:44.078112 9929 leader_election.cc:270] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0517 06:44:44.079687 9933 raft_consensus.cc:457] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } unsafe_config_change: true I0517 06:44:44.080054 9933 leader_election.cc:270] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0517 06:44:44.080467 9934 raft_consensus.cc:2704] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Snoozing failure detection for 1.553s (election complete) I0517 06:44:44.080721 9934 raft_consensus.cc:2511] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 3 FOLLOWER]: Leader election won for term 3 I0517 06:44:44.081099 9934 raft_consensus.cc:571] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 3 LEADER]: Becoming Leader. State: Replica: 44dc106a097740c3b1e984dd2ba3d664, State: Running, Role: LEADER I0517 06:44:44.081452 9935 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: Snoozing failure detection for 1.971s (election complete) I0517 06:44:44.081460 9934 consensus_queue.cc:204] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.3, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } unsafe_config_change: true I0517 06:44:44.081740 9935 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 2 FOLLOWER]: Leader election won for term 2 I0517 06:44:44.085624 9935 raft_consensus.cc:571] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [term 2 LEADER]: Becoming Leader. State: Replica: 952ce954d44a4751b333513bc1e723ec, State: Running, Role: LEADER I0517 06:44:44.086172 9935 consensus_queue.cc:204] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } I0517 06:44:44.089340 9934 raft_consensus.cc:2658] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 3 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4d50d25c58304e38b0d506069d0002ff (127.4.96.4) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "120b69a6c7e24e98ad83d42c78805f4e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 41524 } } } I0517 06:44:44.091886 9937 sys_catalog.cc:335] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "952ce954d44a4751b333513bc1e723ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } } I0517 06:44:44.092344 9937 sys_catalog.cc:338] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: This master's current role is: LEADER W0517 06:44:44.094930 9934 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35679 (request call id 30) took 3839ms (client timeout 2000). W0517 06:44:44.095755 9934 rpcz_store.cc:255] Trace: 0517 06:44:40.255163 (+ 0us) service_pool.cc:162] Inserting onto call queue 0517 06:44:40.255247 (+ 84us) service_pool.cc:221] Handling call 0517 06:44:40.257528 (+ 2281us) log.cc:582] Serialized 364 byte log entry 0517 06:44:44.094877 (+3837349us) inbound_call.cc:162] Queueing success response Metrics: {"dns_us":35,"raft.queue_time_us":131,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":278} I0517 06:44:44.096738 9934 raft_consensus.cc:2658] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 3 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1) evicted, VOTER 4d50d25c58304e38b0d506069d0002ff (127.4.96.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } unsafe_config_change: true } I0517 06:44:44.093228 9929 sys_catalog.cc:335] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "952ce954d44a4751b333513bc1e723ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } } I0517 06:44:44.160971 9929 sys_catalog.cc:338] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: This master's current role is: LEADER I0517 06:44:44.162992 9929 sys_catalog.cc:422] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: configured and running, proceeding with master startup. I0517 06:44:44.100561 9936 sys_catalog.cc:335] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 952ce954d44a4751b333513bc1e723ec. Latest consensus state: current_term: 2 leader_uuid: "952ce954d44a4751b333513bc1e723ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "952ce954d44a4751b333513bc1e723ec" member_type: VOTER } } I0517 06:44:44.165218 9936 sys_catalog.cc:338] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec [sys.catalog]: This master's current role is: LEADER I0517 06:44:44.168092 9867 master_main.cc:86] Master server successfully started. I0517 06:44:44.175535 9943 catalog_manager.cc:1045] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: acquiring CA information for follower catalog manager: success I0517 06:44:44.178366 9943 catalog_manager.cc:1073] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0517 06:44:44.178815 9942 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:44:44.180224 9896 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" instance_seqno: 1526539478015491) as {username='slave'} at 127.4.96.2:59833; Asking this server to re-register. I0517 06:44:44.181641 9942 catalog_manager.cc:321] Loaded metadata for table TestTable [id=6eafaf0c70a344ef85e470ded1fd4cab] I0517 06:44:44.181771 9430 heartbeater.cc:426] Registering TS with master... I0517 06:44:44.183832 9896 ts_manager.cc:81] Registered new tserver with Master: 44dc106a097740c3b1e984dd2ba3d664 (127.4.96.2:42545) I0517 06:44:44.184193 9942 catalog_manager.cc:375] Loaded metadata for tablet 04d150767ee548d6b6d776284666ac51 (table TestTable [id=6eafaf0c70a344ef85e470ded1fd4cab]) I0517 06:44:44.184739 9942 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:44:44.186514 9942 catalog_manager.cc:1021] Loading token signing keys... I0517 06:44:44.188354 9942 catalog_manager.cc:3973] T 00000000000000000000000000000000 P 952ce954d44a4751b333513bc1e723ec: Loaded TSK: 0 I0517 06:44:44.494961 9896 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" instance_seqno: 1526539479428866) as {username='slave'} at 127.4.96.5:59610; Asking this server to re-register. I0517 06:44:44.495836 9803 heartbeater.cc:426] Registering TS with master... I0517 06:44:44.496085 9803 heartbeater.cc:475] Master 127.4.96.62:40202 requested a full tablet report, sending... I0517 06:44:44.497221 9896 ts_manager.cc:81] Registered new tserver with Master: e59d0eda53f7443bb8b30c8f84fa5a37 (127.4.96.5:33454) I0517 06:44:44.664988 9554 heartbeater.cc:346] Connected to a master server at 127.4.96.62:40202 I0517 06:44:44.666307 9896 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "101a15d6048446abbf30c04b2944d96d" instance_seqno: 1526539478596007) as {username='slave'} at 127.4.96.3:59449; Asking this server to re-register. I0517 06:44:44.667142 9554 heartbeater.cc:426] Registering TS with master... I0517 06:44:44.667387 9554 heartbeater.cc:475] Master 127.4.96.62:40202 requested a full tablet report, sending... I0517 06:44:44.668645 9896 ts_manager.cc:81] Registered new tserver with Master: 101a15d6048446abbf30c04b2944d96d (127.4.96.3:39331) I0517 06:44:46.187237 9430 heartbeater.cc:467] Master 127.4.96.62:40202 was elected leader, sending a full tablet report... I0517 06:44:46.189090 9896 catalog_manager.cc:3686] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 reported cstate change: config changed from index -1 to 3, term changed from 1 to 3, VOTER 120b69a6c7e24e98ad83d42c78805f4e (127.4.96.1) evicted, VOTER 4d50d25c58304e38b0d506069d0002ff (127.4.96.4) evicted. New cstate: current_term: 3 leader_uuid: "44dc106a097740c3b1e984dd2ba3d664" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:44:46.203574 9896 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff with a delay of 26 ms (attempt = 1) W0517 06:44:46.203788 9896 catalog_manager.cc:3845] Failed to send 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff: Not found: failed to reset TS proxy: Could not find TS for UUID 4d50d25c58304e38b0d506069d0002ff I0517 06:44:46.203997 9896 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e with a delay of 21 ms (attempt = 1) W0517 06:44:46.204143 9896 catalog_manager.cc:3845] Failed to send 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e: Not found: failed to reset TS proxy: Could not find TS for UUID 120b69a6c7e24e98ad83d42c78805f4e I0517 06:44:46.204536 9896 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 04d150767ee548d6b6d776284666ac51 (attempt 1) I0517 06:44:46.214396 9387 consensus_queue.cc:204] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:46.217510 9957 raft_consensus.cc:2658] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, NON_VOTER e59d0eda53f7443bb8b30c8f84fa5a37 (127.4.96.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:46.221420 9877 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 04d150767ee548d6b6d776284666ac51 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0517 06:44:46.223359 9896 catalog_manager.cc:3686] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 reported cstate change: config changed from index 3 to 5, NON_VOTER e59d0eda53f7443bb8b30c8f84fa5a37 (127.4.96.5) added. New cstate: current_term: 3 leader_uuid: "44dc106a097740c3b1e984dd2ba3d664" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0517 06:44:46.225507 9878 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e with a delay of 50 ms (attempt = 2) W0517 06:44:46.225741 9878 catalog_manager.cc:2810] Async tablet task 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e failed: Not found: failed to reset TS proxy: Could not find TS for UUID 120b69a6c7e24e98ad83d42c78805f4e I0517 06:44:46.227927 9896 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 04d150767ee548d6b6d776284666ac51 (attempt 1) I0517 06:44:46.229614 9387 consensus_peers.cc:456] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer e59d0eda53f7443bb8b30c8f84fa5a37 (127.4.96.5:33454): Closing peer: e59d0eda53f7443bb8b30c8f84fa5a37 I0517 06:44:46.230237 9876 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff with a delay of 43 ms (attempt = 2) W0517 06:44:46.230397 9876 catalog_manager.cc:2810] Async tablet task 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4d50d25c58304e38b0d506069d0002ff I0517 06:44:46.229995 9387 consensus_queue.cc:204] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: NON_VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:46.232614 9957 raft_consensus.cc:2658] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER 101a15d6048446abbf30c04b2944d96d (127.4.96.3) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: NON_VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:46.238088 9877 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 04d150767ee548d6b6d776284666ac51 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0517 06:44:46.239468 9896 catalog_manager.cc:3686] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 reported cstate change: config changed from index 5 to 6, NON_VOTER 101a15d6048446abbf30c04b2944d96d (127.4.96.3) added. New cstate: current_term: 3 leader_uuid: "44dc106a097740c3b1e984dd2ba3d664" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: NON_VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } W0517 06:44:46.242403 9319 consensus_peers.cc:434] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer e59d0eda53f7443bb8b30c8f84fa5a37 (127.4.96.5:33454): Couldn't send request to peer e59d0eda53f7443bb8b30c8f84fa5a37 for tablet 04d150767ee548d6b6d776284666ac51. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04d150767ee548d6b6d776284666ac51. Retrying in the next heartbeat period. Already tried 1 times. W0517 06:44:46.244956 9319 consensus_peers.cc:434] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer 101a15d6048446abbf30c04b2944d96d (127.4.96.3:39331): Couldn't send request to peer 101a15d6048446abbf30c04b2944d96d for tablet 04d150767ee548d6b6d776284666ac51. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04d150767ee548d6b6d776284666ac51. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:46.273819 9876 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff with a delay of 106 ms (attempt = 3) W0517 06:44:46.274018 9876 catalog_manager.cc:2810] Async tablet task 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4d50d25c58304e38b0d506069d0002ff I0517 06:44:46.276212 9878 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e with a delay of 96 ms (attempt = 3) W0517 06:44:46.276381 9878 catalog_manager.cc:2810] Async tablet task 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e failed: Not found: failed to reset TS proxy: Could not find TS for UUID 120b69a6c7e24e98ad83d42c78805f4e I0517 06:44:46.373014 9878 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e with a delay of 174 ms (attempt = 4) W0517 06:44:46.373217 9878 catalog_manager.cc:2810] Async tablet task 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e failed: Not found: failed to reset TS proxy: Could not find TS for UUID 120b69a6c7e24e98ad83d42c78805f4e I0517 06:44:46.380611 9876 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff with a delay of 157 ms (attempt = 4) W0517 06:44:46.380779 9876 catalog_manager.cc:2810] Async tablet task 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4d50d25c58304e38b0d506069d0002ff I0517 06:44:46.538616 9876 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff with a delay of 277 ms (attempt = 5) W0517 06:44:46.538790 9876 catalog_manager.cc:2810] Async tablet task 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4d50d25c58304e38b0d506069d0002ff I0517 06:44:46.548238 9878 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e with a delay of 276 ms (attempt = 5) W0517 06:44:46.548569 9878 catalog_manager.cc:2810] Async tablet task 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e failed: Not found: failed to reset TS proxy: Could not find TS for UUID 120b69a6c7e24e98ad83d42c78805f4e I0517 06:44:46.802426 9964 ts_tablet_manager.cc:669] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: Initiating tablet copy from peer 44dc106a097740c3b1e984dd2ba3d664 (127.4.96.2:42545) I0517 06:44:46.802700 9964 tablet_copy_client.cc:239] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.2:42545 I0517 06:44:46.811193 9407 tablet_copy_service.cc:133] P 44dc106a097740c3b1e984dd2ba3d664: Received BeginTabletCopySession request for tablet 04d150767ee548d6b6d776284666ac51 from peer e59d0eda53f7443bb8b30c8f84fa5a37 ({username='slave'} at 127.4.96.5:56305) I0517 06:44:46.811419 9407 tablet_copy_service.cc:154] P 44dc106a097740c3b1e984dd2ba3d664: Beginning new tablet copy session on tablet 04d150767ee548d6b6d776284666ac51 from peer e59d0eda53f7443bb8b30c8f84fa5a37 at {username='slave'} at 127.4.96.5:56305: session id = e59d0eda53f7443bb8b30c8f84fa5a37-04d150767ee548d6b6d776284666ac51 I0517 06:44:46.814851 9407 tablet_copy_source_session.cc:202] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:44:46.816862 9876 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff with a delay of 546 ms (attempt = 6) W0517 06:44:46.817081 9876 catalog_manager.cc:2810] Async tablet task 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4d50d25c58304e38b0d506069d0002ff I0517 06:44:46.817847 9964 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 04d150767ee548d6b6d776284666ac51. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:46.825816 9878 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e with a delay of 534 ms (attempt = 6) W0517 06:44:46.826076 9878 catalog_manager.cc:2810] Async tablet task 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e failed: Not found: failed to reset TS proxy: Could not find TS for UUID 120b69a6c7e24e98ad83d42c78805f4e I0517 06:44:46.828826 9964 ts_tablet_manager.cc:1173] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:44:46.829160 9964 tablet_copy_client.cc:565] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: tablet copy: Starting download of 0 data blocks... I0517 06:44:46.829387 9964 tablet_copy_client.cc:528] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: tablet copy: Starting download of 1 WAL segments... I0517 06:44:46.830508 9967 ts_tablet_manager.cc:669] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: Initiating tablet copy from peer 44dc106a097740c3b1e984dd2ba3d664 (127.4.96.2:42545) I0517 06:44:46.830785 9967 tablet_copy_client.cc:239] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.2:42545 I0517 06:44:46.831496 9964 tablet_copy_client.cc:404] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:44:46.835175 9964 ts_tablet_manager.cc:1028] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: Bootstrapping tablet I0517 06:44:46.835651 9964 tablet_bootstrap.cc:436] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: Bootstrap starting. I0517 06:44:46.841125 9407 tablet_copy_service.cc:133] P 44dc106a097740c3b1e984dd2ba3d664: Received BeginTabletCopySession request for tablet 04d150767ee548d6b6d776284666ac51 from peer 101a15d6048446abbf30c04b2944d96d ({username='slave'} at 127.4.96.3:50774) I0517 06:44:46.841336 9407 tablet_copy_service.cc:154] P 44dc106a097740c3b1e984dd2ba3d664: Beginning new tablet copy session on tablet 04d150767ee548d6b6d776284666ac51 from peer 101a15d6048446abbf30c04b2944d96d at {username='slave'} at 127.4.96.3:50774: session id = 101a15d6048446abbf30c04b2944d96d-04d150767ee548d6b6d776284666ac51 I0517 06:44:46.841967 9964 log.cc:520] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: Log is configured to *not* fsync() on all Append() calls I0517 06:44:46.844169 9407 tablet_copy_source_session.cc:202] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:44:46.846772 9967 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 04d150767ee548d6b6d776284666ac51. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:46.849611 9964 tablet_bootstrap.cc:436] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:46.850208 9964 tablet_bootstrap.cc:436] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: Bootstrap complete. I0517 06:44:46.850630 9964 ts_tablet_manager.cc:1045] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: Time spent bootstrapping tablet: real 0.015s user 0.016s sys 0.000s I0517 06:44:46.862818 9967 ts_tablet_manager.cc:1173] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:44:46.863265 9967 tablet_copy_client.cc:565] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: tablet copy: Starting download of 0 data blocks... I0517 06:44:46.863543 9967 tablet_copy_client.cc:528] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: tablet copy: Starting download of 1 WAL segments... I0517 06:44:46.864934 9964 raft_consensus.cc:303] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: NON_VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:46.865504 9967 tablet_copy_client.cc:404] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:44:46.865563 9964 raft_consensus.cc:604] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: e59d0eda53f7443bb8b30c8f84fa5a37, State: Initialized, Role: LEARNER I0517 06:44:46.865990 9964 consensus_queue.cc:227] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: NON_VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:46.868530 9964 ts_tablet_manager.cc:1073] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s I0517 06:44:46.869436 9967 ts_tablet_manager.cc:1028] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: Bootstrapping tablet I0517 06:44:46.869664 9407 tablet_copy_service.cc:323] P 44dc106a097740c3b1e984dd2ba3d664: Request end of tablet copy session e59d0eda53f7443bb8b30c8f84fa5a37-04d150767ee548d6b6d776284666ac51 received from {username='slave'} at 127.4.96.5:56305 I0517 06:44:46.869765 9967 tablet_bootstrap.cc:436] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: Bootstrap starting. I0517 06:44:46.869874 9407 tablet_copy_service.cc:411] P 44dc106a097740c3b1e984dd2ba3d664: Ending tablet copy session e59d0eda53f7443bb8b30c8f84fa5a37-04d150767ee548d6b6d776284666ac51 on tablet 04d150767ee548d6b6d776284666ac51 with peer e59d0eda53f7443bb8b30c8f84fa5a37 I0517 06:44:46.875082 9967 log.cc:520] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: Log is configured to *not* fsync() on all Append() calls I0517 06:44:46.881294 9967 tablet_bootstrap.cc:436] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:46.881745 9967 tablet_bootstrap.cc:436] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: Bootstrap complete. I0517 06:44:46.882059 9967 ts_tablet_manager.cc:1045] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: Time spent bootstrapping tablet: real 0.012s user 0.016s sys 0.000s I0517 06:44:46.891218 9967 raft_consensus.cc:303] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: NON_VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:46.891757 9967 raft_consensus.cc:604] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 101a15d6048446abbf30c04b2944d96d, State: Initialized, Role: LEARNER I0517 06:44:46.892061 9967 consensus_queue.cc:227] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: NON_VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:46.894093 9967 ts_tablet_manager.cc:1073] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s I0517 06:44:46.894997 9407 tablet_copy_service.cc:323] P 44dc106a097740c3b1e984dd2ba3d664: Request end of tablet copy session 101a15d6048446abbf30c04b2944d96d-04d150767ee548d6b6d776284666ac51 received from {username='slave'} at 127.4.96.3:50774 I0517 06:44:46.895170 9407 tablet_copy_service.cc:411] P 44dc106a097740c3b1e984dd2ba3d664: Ending tablet copy session 101a15d6048446abbf30c04b2944d96d-04d150767ee548d6b6d776284666ac51 on tablet 04d150767ee548d6b6d776284666ac51 with peer 101a15d6048446abbf30c04b2944d96d W0517 06:44:46.955510 9804 tablet.cc:1678] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:44:47.124526 9555 tablet.cc:1678] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:47.185942 9760 raft_consensus.cc:1002] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0517 06:44:47.203300 9511 raft_consensus.cc:1002] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0517 06:44:47.361330 9878 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e with a delay of 1051 ms (attempt = 7) W0517 06:44:47.361522 9878 catalog_manager.cc:2810] Async tablet task 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=120b69a6c7e24e98ad83d42c78805f4e failed: Not found: failed to reset TS proxy: Could not find TS for UUID 120b69a6c7e24e98ad83d42c78805f4e I0517 06:44:47.363677 9876 catalog_manager.cc:2791] Scheduling retry of 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff with a delay of 1058 ms (attempt = 7) W0517 06:44:47.363844 9876 catalog_manager.cc:2810] Async tablet task 04d150767ee548d6b6d776284666ac51 Delete Tablet RPC for TS=4d50d25c58304e38b0d506069d0002ff failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4d50d25c58304e38b0d506069d0002ff I0517 06:44:47.628372 9974 raft_consensus.cc:866] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: attempting to promote NON_VOTER 101a15d6048446abbf30c04b2944d96d to VOTER I0517 06:44:47.629037 9974 consensus_peers.cc:456] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer 101a15d6048446abbf30c04b2944d96d (127.4.96.3:39331): Closing peer: 101a15d6048446abbf30c04b2944d96d I0517 06:44:47.629195 9974 consensus_peers.cc:456] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer e59d0eda53f7443bb8b30c8f84fa5a37 (127.4.96.5:33454): Closing peer: e59d0eda53f7443bb8b30c8f84fa5a37 I0517 06:44:47.629628 9974 consensus_queue.cc:204] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: false } } unsafe_config_change: true I0517 06:44:47.632863 9511 raft_consensus.cc:1057] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d [term 3 LEARNER]: Refusing update from remote peer 44dc106a097740c3b1e984dd2ba3d664: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:44:47.633075 9760 raft_consensus.cc:1057] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37 [term 3 LEARNER]: Refusing update from remote peer 44dc106a097740c3b1e984dd2ba3d664: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:44:47.633877 9974 consensus_queue.cc:963] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:44:47.634389 9963 consensus_queue.cc:963] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:44:47.639403 9973 raft_consensus.cc:829] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 3 LEADER]: attempt to promote peer e59d0eda53f7443bb8b30c8f84fa5a37: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing. I0517 06:44:47.639670 9963 raft_consensus.cc:2658] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 101a15d6048446abbf30c04b2944d96d (127.4.96.3) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:47.640671 9511 raft_consensus.cc:2658] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 101a15d6048446abbf30c04b2944d96d (127.4.96.3) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:47.645138 9760 raft_consensus.cc:2658] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37 [term 3 LEARNER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 101a15d6048446abbf30c04b2944d96d (127.4.96.3) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:47.645943 9896 catalog_manager.cc:3686] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 reported cstate change: config changed from index 6 to 7, 101a15d6048446abbf30c04b2944d96d (127.4.96.3) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "44dc106a097740c3b1e984dd2ba3d664" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:44:47.651805 9974 raft_consensus.cc:866] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664: attempting to promote NON_VOTER e59d0eda53f7443bb8b30c8f84fa5a37 to VOTER I0517 06:44:47.652362 9974 consensus_peers.cc:456] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer 101a15d6048446abbf30c04b2944d96d (127.4.96.3:39331): Closing peer: 101a15d6048446abbf30c04b2944d96d I0517 06:44:47.652492 9974 consensus_peers.cc:456] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 -> Peer e59d0eda53f7443bb8b30c8f84fa5a37 (127.4.96.5:33454): Closing peer: e59d0eda53f7443bb8b30c8f84fa5a37 I0517 06:44:47.652787 9974 consensus_queue.cc:204] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: false } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: false } } unsafe_config_change: true I0517 06:44:47.655706 9760 raft_consensus.cc:1057] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37 [term 3 LEARNER]: Refusing update from remote peer 44dc106a097740c3b1e984dd2ba3d664: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0517 06:44:47.655808 9511 raft_consensus.cc:1057] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d [term 3 FOLLOWER]: Refusing update from remote peer 44dc106a097740c3b1e984dd2ba3d664: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0517 06:44:47.656786 9963 consensus_queue.cc:963] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0517 06:44:47.657336 9973 consensus_queue.cc:963] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0517 06:44:47.661687 9973 raft_consensus.cc:2658] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, e59d0eda53f7443bb8b30c8f84fa5a37 (127.4.96.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: false } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:47.664556 9511 raft_consensus.cc:2658] T 04d150767ee548d6b6d776284666ac51 P 101a15d6048446abbf30c04b2944d96d [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, e59d0eda53f7443bb8b30c8f84fa5a37 (127.4.96.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: false } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:47.667590 9894 catalog_manager.cc:3686] T 04d150767ee548d6b6d776284666ac51 P 44dc106a097740c3b1e984dd2ba3d664 reported cstate change: config changed from index 7 to 8, e59d0eda53f7443bb8b30c8f84fa5a37 (127.4.96.5) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "44dc106a097740c3b1e984dd2ba3d664" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:44:47.667587 9760 raft_consensus.cc:2658] T 04d150767ee548d6b6d776284666ac51 P e59d0eda53f7443bb8b30c8f84fa5a37 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, e59d0eda53f7443bb8b30c8f84fa5a37 (127.4.96.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 42545 } } peers { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 33454 } attrs { promote: false } } peers { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39331 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:47.790179 4480 kudu-admin-test.cc:800] Waiting for Master to see new config... I0517 06:44:47.800117 4480 kudu-admin-test.cc:805] Tablet locations: tablet_id: "04d150767ee548d6b6d776284666ac51" replicas { ts_info { permanent_uuid: "44dc106a097740c3b1e984dd2ba3d664" rpc_addresses { host: "127.4.96.2" port: 42545 } } role: LEADER } replicas { ts_info { permanent_uuid: "e59d0eda53f7443bb8b30c8f84fa5a37" rpc_addresses { host: "127.4.96.5" port: 33454 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "101a15d6048446abbf30c04b2944d96d" rpc_addresses { host: "127.4.96.3" port: 39331 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0517 06:44:47.801460 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 9309 W0517 06:44:47.822443 9691 connection.cc:512] client connection to 127.4.96.2:42545 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:42545): Connection reset by peer (error 104) W0517 06:44:47.822485 9877 connection.cc:512] server connection from 127.4.96.2:59833 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:59833): Connection reset by peer (error 104) W0517 06:44:47.822535 9111 connection.cc:512] client connection to 127.4.96.2:42545 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:42545): Connection reset by peer (error 104) W0517 06:44:47.823567 9877 connection.cc:512] client connection to 127.4.96.2:42545 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:42545): Connection reset by peer (error 104) W0517 06:44:47.823648 9692 connection.cc:512] server connection from 127.4.96.2:36917 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:36917): Connection reset by peer (error 104) W0517 06:44:47.823683 9108 connection.cc:512] client connection to 127.4.96.2:42545 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:42545): Connection reset by peer (error 104) W0517 06:44:47.824618 9442 connection.cc:512] client connection to 127.4.96.2:42545 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:42545): Connection reset by peer (error 104) I0517 06:44:47.824820 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 9433 W0517 06:44:47.851645 9878 connection.cc:512] server connection from 127.4.96.3:59449 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:59449): Connection reset by peer (error 104) W0517 06:44:47.851662 9111 connection.cc:512] client connection to 127.4.96.3:39331 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:39331): Connection reset by peer (error 104) W0517 06:44:47.851732 9109 connection.cc:512] client connection to 127.4.96.3:39331 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:39331): Connection reset by peer (error 104) I0517 06:44:47.862156 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 9682 W0517 06:44:47.880275 9111 connection.cc:512] client connection to 127.4.96.5:33454 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:33454): Connection reset by peer (error 104) W0517 06:44:47.880306 9877 connection.cc:512] server connection from 127.4.96.5:59610 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:59610): Connection reset by peer (error 104) I0517 06:44:47.880436 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 9867 W0517 06:44:47.908313 9111 connection.cc:512] client connection to 127.4.96.62:40202 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:40202): Connection reset by peer (error 104) [ OK ] AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig (11349 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig I0517 06:44:47.923247 4480 test_util.cc:200] Using random seed: 736742427 I0517 06:44:47.926265 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:44:47.926362 4480 ts_itest-base.cc:114] -------------- I0517 06:44:47.926456 4480 ts_itest-base.cc:115] 5 tablet servers I0517 06:44:47.926548 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:44:47.926643 4480 ts_itest-base.cc:117] -------------- I0517 06:44:47.928867 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:41305 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:48.041158 9990 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:48.041476 9990 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:48.050319 9990 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:44:48.050436 9990 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:48.050535 9990 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:44:48.050647 9990 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:44:48.064491 9990 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:41305 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:48.072090 9990 master_main.cc:80] Initializing master server... I0517 06:44:48.073596 9990 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 143319us I0517 06:44:48.074359 9990 fs_manager.cc:260] Metadata directory not provided I0517 06:44:48.074479 9990 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:48.074662 9990 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:44:48.074761 9990 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:48.081892 9990 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "dba82cf69c2a4538a8be63dd114489da" format_stamp: "Formatted at 2018-05-17 06:44:48 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:48.082550 9990 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "dba82cf69c2a4538a8be63dd114489da" format_stamp: "Formatted at 2018-05-17 06:44:48 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:48.091935 9990 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:44:48.092990 9990 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:48.093431 9990 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:48.099750 9990 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:44:48.099856 9990 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "dba82cf69c2a4538a8be63dd114489da" format_stamp: "Formatted at 2018-05-17 06:44:48 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:48.100039 9990 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:48.269795 9990 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:48.271514 9990 master_main.cc:83] Starting Master server... I0517 06:44:48.311027 9990 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:41305 I0517 06:44:48.311858 9990 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:44:48.311954 9990 webserver.cc:184] Document root disabled I0517 06:44:48.313150 9990 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:56915/ I0517 06:44:48.313832 9990 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:44:48.316416 10051 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:48.321378 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 9990 I0517 06:44:48.322741 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:41305 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:48.332361 10051 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: Bootstrap starting. I0517 06:44:48.334628 10051 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: No blocks or log segments found. Creating new log. I0517 06:44:48.335773 10051 log.cc:520] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: Log is configured to *not* fsync() on all Append() calls I0517 06:44:48.338057 10051 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: No bootstrap required, opened a new log I0517 06:44:48.349637 10051 raft_consensus.cc:303] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } I0517 06:44:48.349973 10051 raft_consensus.cc:329] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:48.350108 10051 raft_consensus.cc:604] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba82cf69c2a4538a8be63dd114489da, State: Initialized, Role: FOLLOWER I0517 06:44:48.350539 10051 consensus_queue.cc:227] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } I0517 06:44:48.350886 10051 raft_consensus.cc:343] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:44:48.351033 10051 raft_consensus.cc:435] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:44:48.351177 10051 raft_consensus.cc:2704] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: Snoozing failure detection for 1.610s (starting election) I0517 06:44:48.351315 10051 raft_consensus.cc:2756] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:48.353920 10051 raft_consensus.cc:457] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } I0517 06:44:48.354365 10051 leader_election.cc:270] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:44:48.355962 10054 raft_consensus.cc:2704] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: Snoozing failure detection for 2.041s (election complete) I0517 06:44:48.356242 10054 raft_consensus.cc:2511] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:44:48.362478 10054 raft_consensus.cc:571] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 1 LEADER]: Becoming Leader. State: Replica: dba82cf69c2a4538a8be63dd114489da, State: Running, Role: LEADER I0517 06:44:48.362903 10054 consensus_queue.cc:204] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } I0517 06:44:48.365963 10051 sys_catalog.cc:335] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "dba82cf69c2a4538a8be63dd114489da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } } I0517 06:44:48.366374 10051 sys_catalog.cc:338] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: This master's current role is: LEADER I0517 06:44:48.368402 10051 sys_catalog.cc:422] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: configured and running, proceeding with master startup. WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:48.435175 10052 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:48.435528 10052 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:44:48.368618 10058 catalog_manager.cc:1001] Loading table and tablet metadata into memory... W0517 06:44:48.443696 10052 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:48.444018 10052 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:44:48.369415 10056 sys_catalog.cc:335] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dba82cf69c2a4538a8be63dd114489da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } } I0517 06:44:48.455238 10056 sys_catalog.cc:338] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: This master's current role is: LEADER I0517 06:44:48.369421 10055 sys_catalog.cc:335] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: SysCatalogTable state changed. Reason: New leader dba82cf69c2a4538a8be63dd114489da. Latest consensus state: current_term: 1 leader_uuid: "dba82cf69c2a4538a8be63dd114489da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } } I0517 06:44:48.456851 10055 sys_catalog.cc:338] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: This master's current role is: LEADER I0517 06:44:48.457696 10059 catalog_manager.cc:504] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0517 06:44:48.459276 9990 master_main.cc:86] Master server successfully started. I0517 06:44:48.459933 10052 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:41305 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:48.462318 10058 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:44:48.472658 10052 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:48.475021 10052 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 143319us I0517 06:44:48.475620 10052 fs_manager.cc:260] Metadata directory not provided I0517 06:44:48.475785 10052 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:48.476047 10052 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:44:48.476224 10052 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:48.488662 10052 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "4a56ce22b67e4bd795817145c47d2a15" format_stamp: "Formatted at 2018-05-17 06:44:48 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:48.489610 10052 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "4a56ce22b67e4bd795817145c47d2a15" format_stamp: "Formatted at 2018-05-17 06:44:48 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:48.503496 10052 fs_manager.cc:495] Time spent creating directory manager: real 0.013s user 0.012s sys 0.000s I0517 06:44:48.505249 10052 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:48.505802 10052 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:48.528409 10052 fs_manager.cc:417] Time spent opening block manager: real 0.014s user 0.004s sys 0.008s I0517 06:44:48.528592 10052 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "4a56ce22b67e4bd795817145c47d2a15" format_stamp: "Formatted at 2018-05-17 06:44:48 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:48.528906 10052 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:48.539875 10058 catalog_manager.cc:899] Generated new certificate authority record I0517 06:44:48.542027 10058 catalog_manager.cc:1021] Loading token signing keys... I0517 06:44:48.569006 10058 catalog_manager.cc:3962] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: Generated new TSK 0 I0517 06:44:48.762958 10052 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:48.765486 10052 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:48.766619 10052 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:48.868365 10052 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:47178 I0517 06:44:48.869268 10052 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:44:48.869374 10052 webserver.cc:184] Document root disabled I0517 06:44:48.870777 10052 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:60279/ I0517 06:44:48.871484 10052 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:44:48.875811 10052 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:48.879062 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 10052 I0517 06:44:48.880311 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:41305 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:48.902405 10183 heartbeater.cc:346] Connected to a master server at 127.4.96.62:41305 I0517 06:44:48.902698 10183 heartbeater.cc:426] Registering TS with master... I0517 06:44:48.903275 10183 heartbeater.cc:475] Master 127.4.96.62:41305 requested a full tablet report, sending... I0517 06:44:48.905222 10018 ts_manager.cc:81] Registered new tserver with Master: 4a56ce22b67e4bd795817145c47d2a15 (127.4.96.1:47178) I0517 06:44:48.907727 10018 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:53626 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:48.990659 10187 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:48.990983 10187 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:48.998953 10187 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:48.999280 10187 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:44:49.012135 10187 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:41305 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:49.020308 10187 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:49.022092 10187 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 143819us I0517 06:44:49.022521 10187 fs_manager.cc:260] Metadata directory not provided I0517 06:44:49.022647 10187 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:49.022857 10187 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:44:49.022979 10187 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:49.030580 10187 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "28ba6c7ee5464c6db38265b65d3fee0f" format_stamp: "Formatted at 2018-05-17 06:44:49 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:49.031242 10187 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "28ba6c7ee5464c6db38265b65d3fee0f" format_stamp: "Formatted at 2018-05-17 06:44:49 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:49.040731 10187 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:44:49.041837 10187 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:49.042250 10187 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:49.048866 10187 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0517 06:44:49.048979 10187 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "28ba6c7ee5464c6db38265b65d3fee0f" format_stamp: "Formatted at 2018-05-17 06:44:49 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:49.049185 10187 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:49.257038 10187 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:49.259613 10187 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:49.260700 10187 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:49.363373 10187 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:55398 I0517 06:44:49.364264 10187 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:44:49.364368 10187 webserver.cc:184] Document root disabled I0517 06:44:49.365715 10187 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:58517/ I0517 06:44:49.366420 10187 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:44:49.368129 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 10187 I0517 06:44:49.369298 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:41305 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:49.398217 10187 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:49.421730 10308 heartbeater.cc:346] Connected to a master server at 127.4.96.62:41305 I0517 06:44:49.422056 10308 heartbeater.cc:426] Registering TS with master... I0517 06:44:49.422639 10308 heartbeater.cc:475] Master 127.4.96.62:41305 requested a full tablet report, sending... I0517 06:44:49.424437 10018 ts_manager.cc:81] Registered new tserver with Master: 28ba6c7ee5464c6db38265b65d3fee0f (127.4.96.2:55398) I0517 06:44:49.426455 10018 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:54722 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:49.489171 10311 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:49.489504 10311 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:49.497674 10311 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:49.498044 10311 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:44:49.511085 10311 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:41305 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:49.519493 10311 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:49.521273 10311 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 143819us I0517 06:44:49.521718 10311 fs_manager.cc:260] Metadata directory not provided I0517 06:44:49.521838 10311 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:49.522069 10311 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:44:49.522188 10311 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:49.529904 10311 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "15ac5271aef7440e87dea2eaa9321ca2" format_stamp: "Formatted at 2018-05-17 06:44:49 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:49.530609 10311 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "15ac5271aef7440e87dea2eaa9321ca2" format_stamp: "Formatted at 2018-05-17 06:44:49 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:49.540118 10311 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:44:49.541553 10311 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:49.541978 10311 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:49.548491 10311 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:44:49.548612 10311 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "15ac5271aef7440e87dea2eaa9321ca2" format_stamp: "Formatted at 2018-05-17 06:44:49 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:49.549146 10311 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:49.763409 10311 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:49.765959 10311 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:49.767125 10311 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:49.851553 10311 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:33068 I0517 06:44:49.852458 10311 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:44:49.852561 10311 webserver.cc:184] Document root disabled I0517 06:44:49.853755 10311 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:55239/ I0517 06:44:49.854452 10311 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:44:49.857498 10311 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:49.861804 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 10311 I0517 06:44:49.862895 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:41305 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:49.874640 10432 heartbeater.cc:346] Connected to a master server at 127.4.96.62:41305 I0517 06:44:49.874884 10432 heartbeater.cc:426] Registering TS with master... I0517 06:44:49.875334 10432 heartbeater.cc:475] Master 127.4.96.62:41305 requested a full tablet report, sending... I0517 06:44:49.876616 10018 ts_manager.cc:81] Registered new tserver with Master: 15ac5271aef7440e87dea2eaa9321ca2 (127.4.96.3:33068) I0517 06:44:49.877697 10018 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:34962 I0517 06:44:49.910938 10183 heartbeater.cc:467] Master 127.4.96.62:41305 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:49.964838 10435 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:49.965143 10435 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:49.973250 10435 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:49.973562 10435 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:44:49.985993 10435 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:41305 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:49.993885 10435 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:49.995669 10435 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 143819us I0517 06:44:49.996114 10435 fs_manager.cc:260] Metadata directory not provided I0517 06:44:49.996232 10435 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:49.996414 10435 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:44:49.996523 10435 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:50.003877 10435 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "6bdab74f06a741abb63715127aeb2811" format_stamp: "Formatted at 2018-05-17 06:44:49 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:50.004515 10435 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "6bdab74f06a741abb63715127aeb2811" format_stamp: "Formatted at 2018-05-17 06:44:49 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:50.013845 10435 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:44:50.014945 10435 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:50.015347 10435 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:50.021602 10435 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.004s I0517 06:44:50.021709 10435 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "6bdab74f06a741abb63715127aeb2811" format_stamp: "Formatted at 2018-05-17 06:44:49 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:50.021919 10435 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:50.180590 10435 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:50.182896 10435 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:50.183835 10435 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:50.271168 10435 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:60463 I0517 06:44:50.272392 10435 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:44:50.272538 10435 webserver.cc:184] Document root disabled I0517 06:44:50.273991 10435 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:42921/ I0517 06:44:50.274844 10435 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:44:50.277024 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 10435 I0517 06:44:50.277964 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.4.96.5:0 --local_ip_for_outbound_sockets=127.4.96.5 --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:41305 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:50.285131 10435 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:50.302105 10556 heartbeater.cc:346] Connected to a master server at 127.4.96.62:41305 I0517 06:44:50.302338 10556 heartbeater.cc:426] Registering TS with master... I0517 06:44:50.302767 10556 heartbeater.cc:475] Master 127.4.96.62:41305 requested a full tablet report, sending... I0517 06:44:50.304108 10018 ts_manager.cc:81] Registered new tserver with Master: 6bdab74f06a741abb63715127aeb2811 (127.4.96.4:60463) I0517 06:44:50.305191 10018 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:49960 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:50.383057 10558 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:50.383364 10558 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:50.391259 10558 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:50.391571 10558 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.5 I0517 06:44:50.403693 10558 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:41305 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.5 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:50.411510 10558 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:50.413316 10558 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 144319us I0517 06:44:50.413740 10558 fs_manager.cc:260] Metadata directory not provided I0517 06:44:50.413853 10558 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:50.414063 10558 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0517 06:44:50.414173 10558 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:50.421464 10558 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "da1d7cec6f814868a154aa0516cf5098" format_stamp: "Formatted at 2018-05-17 06:44:50 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:50.422097 10558 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "da1d7cec6f814868a154aa0516cf5098" format_stamp: "Formatted at 2018-05-17 06:44:50 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:50.429487 10308 heartbeater.cc:467] Master 127.4.96.62:41305 was elected leader, sending a full tablet report... I0517 06:44:50.431383 10558 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:44:50.432461 10558 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:50.432827 10558 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:50.439134 10558 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.004s I0517 06:44:50.439240 10558 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal uuid: "da1d7cec6f814868a154aa0516cf5098" format_stamp: "Formatted at 2018-05-17 06:44:50 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:50.439447 10558 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:50.641762 10558 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:50.644119 10558 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:50.645062 10558 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:50.728616 10558 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.5:46584 I0517 06:44:50.729470 10558 webserver.cc:173] Starting webserver on 127.4.96.5:0 I0517 06:44:50.729569 10558 webserver.cc:184] Document root disabled I0517 06:44:50.730736 10558 webserver.cc:311] Webserver started. Bound to: http://127.4.96.5:39245/ I0517 06:44:50.731402 10558 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb I0517 06:44:50.734460 10558 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:50.734930 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 10558 I0517 06:44:50.752393 10680 heartbeater.cc:346] Connected to a master server at 127.4.96.62:41305 I0517 06:44:50.752610 10680 heartbeater.cc:426] Registering TS with master... I0517 06:44:50.753028 10680 heartbeater.cc:475] Master 127.4.96.62:41305 requested a full tablet report, sending... I0517 06:44:50.754343 10018 ts_manager.cc:81] Registered new tserver with Master: da1d7cec6f814868a154aa0516cf5098 (127.4.96.5:46584) I0517 06:44:50.755612 10018 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.5:57398 I0517 06:44:50.757915 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:44:50.776273 10018 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60241: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:44:50.828783 10120 tablet_service.cc:759] Processing CreateTablet for tablet 881bb0d82041492bbd4945aa8c43312a (table=TestTable [id=a4d22e57e98a4e1aa50cbd38128cde9c]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:50.829751 10120 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 881bb0d82041492bbd4945aa8c43312a. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:50.835733 10369 tablet_service.cc:759] Processing CreateTablet for tablet 881bb0d82041492bbd4945aa8c43312a (table=TestTable [id=a4d22e57e98a4e1aa50cbd38128cde9c]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:50.836596 10369 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 881bb0d82041492bbd4945aa8c43312a. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:50.838886 10617 tablet_service.cc:759] Processing CreateTablet for tablet 881bb0d82041492bbd4945aa8c43312a (table=TestTable [id=a4d22e57e98a4e1aa50cbd38128cde9c]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:44:50.839687 10617 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 881bb0d82041492bbd4945aa8c43312a. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:50.844883 10120 ts_tablet_manager.cc:1173] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:50.847929 10369 ts_tablet_manager.cc:1173] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:50.855432 10617 ts_tablet_manager.cc:1173] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098: Registered tablet (data state: TABLET_DATA_READY) I0517 06:44:50.857074 10700 ts_tablet_manager.cc:1028] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Bootstrapping tablet I0517 06:44:50.857437 10701 ts_tablet_manager.cc:1028] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2: Bootstrapping tablet I0517 06:44:50.857537 10700 tablet_bootstrap.cc:436] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Bootstrap starting. I0517 06:44:50.857611 10702 ts_tablet_manager.cc:1028] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098: Bootstrapping tablet I0517 06:44:50.857863 10701 tablet_bootstrap.cc:436] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2: Bootstrap starting. I0517 06:44:50.858016 10702 tablet_bootstrap.cc:436] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098: Bootstrap starting. I0517 06:44:50.860194 10702 tablet_bootstrap.cc:587] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098: No blocks or log segments found. Creating new log. I0517 06:44:50.861603 10700 tablet_bootstrap.cc:587] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: No blocks or log segments found. Creating new log. I0517 06:44:50.861672 10701 tablet_bootstrap.cc:587] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2: No blocks or log segments found. Creating new log. I0517 06:44:50.861704 10702 log.cc:520] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098: Log is configured to *not* fsync() on all Append() calls I0517 06:44:50.863190 10701 log.cc:520] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2: Log is configured to *not* fsync() on all Append() calls I0517 06:44:50.863241 10700 log.cc:520] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Log is configured to *not* fsync() on all Append() calls I0517 06:44:50.866415 10701 tablet_bootstrap.cc:436] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2: No bootstrap required, opened a new log I0517 06:44:50.866751 10701 ts_tablet_manager.cc:1045] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s I0517 06:44:50.866781 10700 tablet_bootstrap.cc:436] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: No bootstrap required, opened a new log I0517 06:44:50.867097 10700 ts_tablet_manager.cc:1045] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s I0517 06:44:50.878587 10702 tablet_bootstrap.cc:436] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098: No bootstrap required, opened a new log I0517 06:44:50.878844 10702 ts_tablet_manager.cc:1045] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s I0517 06:44:50.881855 10701 raft_consensus.cc:303] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 33068 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } I0517 06:44:50.882591 10701 raft_consensus.cc:329] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:50.882787 10701 raft_consensus.cc:604] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ac5271aef7440e87dea2eaa9321ca2, State: Initialized, Role: FOLLOWER I0517 06:44:50.883460 10701 consensus_queue.cc:227] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 33068 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } I0517 06:44:50.888329 10700 raft_consensus.cc:303] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 33068 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } I0517 06:44:50.889083 10700 raft_consensus.cc:329] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:50.889297 10700 raft_consensus.cc:604] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a56ce22b67e4bd795817145c47d2a15, State: Initialized, Role: FOLLOWER I0517 06:44:50.889940 10700 consensus_queue.cc:227] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 33068 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } I0517 06:44:50.891640 10702 raft_consensus.cc:303] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 33068 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } I0517 06:44:50.892357 10702 raft_consensus.cc:329] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:50.892552 10702 raft_consensus.cc:604] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1d7cec6f814868a154aa0516cf5098, State: Initialized, Role: FOLLOWER I0517 06:44:50.893157 10702 consensus_queue.cc:227] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 33068 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } I0517 06:44:50.904971 10700 ts_tablet_manager.cc:1073] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Time spent starting tablet: real 0.038s user 0.020s sys 0.008s I0517 06:44:50.906512 10680 heartbeater.cc:467] Master 127.4.96.62:41305 was elected leader, sending a full tablet report... I0517 06:44:50.913341 10702 ts_tablet_manager.cc:1073] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098: Time spent starting tablet: real 0.034s user 0.024s sys 0.004s I0517 06:44:50.879915 10432 heartbeater.cc:467] Master 127.4.96.62:41305 was elected leader, sending a full tablet report... I0517 06:44:50.921537 10701 ts_tablet_manager.cc:1073] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2: Time spent starting tablet: real 0.055s user 0.020s sys 0.000s I0517 06:44:50.958418 10707 raft_consensus.cc:435] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:44:51.020134 10707 raft_consensus.cc:2704] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Snoozing failure detection for 2.040s (starting election) I0517 06:44:51.020375 10707 raft_consensus.cc:457] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 33068 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } I0517 06:44:51.021368 10707 leader_election.cc:243] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 15ac5271aef7440e87dea2eaa9321ca2 (127.4.96.3:33068) I0517 06:44:51.021829 10707 leader_election.cc:243] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer da1d7cec6f814868a154aa0516cf5098 (127.4.96.5:46584) I0517 06:44:51.035436 10389 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "881bb0d82041492bbd4945aa8c43312a" candidate_uuid: "4a56ce22b67e4bd795817145c47d2a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ac5271aef7440e87dea2eaa9321ca2" is_pre_election: true I0517 06:44:51.035894 10389 raft_consensus.cc:2704] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2: Snoozing failure detection for 1.514s (vote granted) I0517 06:44:51.036087 10389 raft_consensus.cc:2219] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a56ce22b67e4bd795817145c47d2a15 in term 0. I0517 06:44:51.036996 10070 leader_election.cc:379] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 15ac5271aef7440e87dea2eaa9321ca2 (127.4.96.3:33068) I0517 06:44:51.037191 10070 leader_election.cc:270] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:44:51.037585 10707 raft_consensus.cc:2704] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Snoozing failure detection for 1.802s (election complete) I0517 06:44:51.037777 10707 raft_consensus.cc:2511] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:44:51.037941 10707 raft_consensus.cc:435] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:44:51.038100 10707 raft_consensus.cc:2704] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Snoozing failure detection for 1.906s (starting election) I0517 06:44:51.038270 10707 raft_consensus.cc:2756] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:51.039685 10637 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "881bb0d82041492bbd4945aa8c43312a" candidate_uuid: "4a56ce22b67e4bd795817145c47d2a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1d7cec6f814868a154aa0516cf5098" is_pre_election: true I0517 06:44:51.040263 10637 raft_consensus.cc:2704] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098: Snoozing failure detection for 1.547s (vote granted) I0517 06:44:51.040513 10637 raft_consensus.cc:2219] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a56ce22b67e4bd795817145c47d2a15 in term 0. I0517 06:44:51.041741 10072 leader_election.cc:379] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [CANDIDATE]: Term 1 pre-election: Vote granted by peer da1d7cec6f814868a154aa0516cf5098 (127.4.96.5:46584) I0517 06:44:51.042440 10707 raft_consensus.cc:457] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 33068 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } I0517 06:44:51.043126 10707 leader_election.cc:243] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [CANDIDATE]: Term 1 election: Requesting vote from peer 15ac5271aef7440e87dea2eaa9321ca2 (127.4.96.3:33068) I0517 06:44:51.043426 10707 leader_election.cc:243] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [CANDIDATE]: Term 1 election: Requesting vote from peer da1d7cec6f814868a154aa0516cf5098 (127.4.96.5:46584) I0517 06:44:51.044235 10389 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "881bb0d82041492bbd4945aa8c43312a" candidate_uuid: "4a56ce22b67e4bd795817145c47d2a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ac5271aef7440e87dea2eaa9321ca2" I0517 06:44:51.044622 10389 raft_consensus.cc:2756] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:51.044576 10637 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "881bb0d82041492bbd4945aa8c43312a" candidate_uuid: "4a56ce22b67e4bd795817145c47d2a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1d7cec6f814868a154aa0516cf5098" I0517 06:44:51.044860 10389 raft_consensus.cc:2704] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2: Snoozing failure detection for 2.136s (vote granted) I0517 06:44:51.044947 10637 raft_consensus.cc:2756] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:51.045172 10637 raft_consensus.cc:2704] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098: Snoozing failure detection for 2.060s (vote granted) I0517 06:44:51.048859 10389 raft_consensus.cc:2219] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a56ce22b67e4bd795817145c47d2a15 in term 1. W0517 06:44:50.985882 10681 tablet.cc:1678] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:51.049343 10637 raft_consensus.cc:2219] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a56ce22b67e4bd795817145c47d2a15 in term 1. I0517 06:44:51.049904 10070 leader_election.cc:379] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [CANDIDATE]: Term 1 election: Vote granted by peer 15ac5271aef7440e87dea2eaa9321ca2 (127.4.96.3:33068) I0517 06:44:51.050143 10070 leader_election.cc:270] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:44:51.050320 10072 leader_election.cc:379] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [CANDIDATE]: Term 1 election: Vote granted by peer da1d7cec6f814868a154aa0516cf5098 (127.4.96.5:46584) I0517 06:44:51.050580 10707 raft_consensus.cc:2704] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Snoozing failure detection for 1.706s (election complete) I0517 06:44:51.050781 10707 raft_consensus.cc:2511] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:44:51.051754 10707 raft_consensus.cc:571] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 1 LEADER]: Becoming Leader. State: Replica: 4a56ce22b67e4bd795817145c47d2a15, State: Running, Role: LEADER I0517 06:44:51.052177 10707 consensus_queue.cc:204] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 33068 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } I0517 06:44:51.056872 10017 catalog_manager.cc:3686] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 reported cstate change: term changed from 0 to 1, leader changed from to 4a56ce22b67e4bd795817145c47d2a15 (127.4.96.1). New cstate: current_term: 1 leader_uuid: "4a56ce22b67e4bd795817145c47d2a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 33068 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } health_report { overall_health: UNKNOWN } } } I0517 06:44:51.103303 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:44:51.105309 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4a56ce22b67e4bd795817145c47d2a15 to finish bootstrapping W0517 06:44:51.109141 10433 tablet.cc:1678] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:51.114295 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 28ba6c7ee5464c6db38265b65d3fee0f to finish bootstrapping I0517 06:44:51.124155 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 15ac5271aef7440e87dea2eaa9321ca2 to finish bootstrapping W0517 06:44:51.128681 10184 tablet.cc:1678] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:51.132746 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6bdab74f06a741abb63715127aeb2811 to finish bootstrapping I0517 06:44:51.142554 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver da1d7cec6f814868a154aa0516cf5098 to finish bootstrapping I0517 06:44:51.150965 4480 kudu-admin-test.cc:836] Waiting for Master to see the current replicas... I0517 06:44:51.153008 4480 kudu-admin-test.cc:843] Tablet locations: tablet_id: "881bb0d82041492bbd4945aa8c43312a" replicas { ts_info { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" rpc_addresses { host: "127.4.96.1" port: 47178 } } role: LEADER } replicas { ts_info { permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" rpc_addresses { host: "127.4.96.3" port: 33068 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" rpc_addresses { host: "127.4.96.5" port: 46584 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0517 06:44:51.307370 10556 heartbeater.cc:467] Master 127.4.96.62:41305 was elected leader, sending a full tablet report... I0517 06:44:51.495240 10637 raft_consensus.cc:1057] T 881bb0d82041492bbd4945aa8c43312a P da1d7cec6f814868a154aa0516cf5098 [term 1 FOLLOWER]: Refusing update from remote peer 4a56ce22b67e4bd795817145c47d2a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:51.496347 10707 consensus_queue.cc:963] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:51.504137 10389 raft_consensus.cc:1057] T 881bb0d82041492bbd4945aa8c43312a P 15ac5271aef7440e87dea2eaa9321ca2 [term 1 FOLLOWER]: Refusing update from remote peer 4a56ce22b67e4bd795817145c47d2a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:44:51.507622 10707 consensus_queue.cc:963] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 33068 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:44:51.515676 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 10558 W0517 06:44:51.532738 10072 connection.cc:512] client connection to 127.4.96.5:46584 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:46584): Connection reset by peer (error 104) W0517 06:44:51.532735 10000 connection.cc:512] client connection to 127.4.96.5:46584 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:46584): Connection reset by peer (error 104) W0517 06:44:51.532768 9989 connection.cc:512] client connection to 127.4.96.5:46584 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:46584): Connection reset by peer (error 104) W0517 06:44:51.533682 10000 connection.cc:512] server connection from 127.4.96.5:57398 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:57398): Connection reset by peer (error 104) W0517 06:44:51.533709 9987 connection.cc:512] client connection to 127.4.96.5:46584 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:46584): Connection reset by peer (error 104) I0517 06:44:51.534763 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 10311 W0517 06:44:51.552459 9998 connection.cc:512] client connection to 127.4.96.3:33068 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:33068): Connection reset by peer (error 104) W0517 06:44:51.552508 9999 connection.cc:512] server connection from 127.4.96.3:34962 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:34962): Connection reset by peer (error 104) W0517 06:44:51.553223 10070 connection.cc:512] client connection to 127.4.96.3:33068 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:33068): Connection reset by peer (error 104) W0517 06:44:51.553270 9989 connection.cc:512] client connection to 127.4.96.3:33068 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:33068): Connection reset by peer (error 104) I0517 06:44:51.553375 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 9990 W0517 06:44:51.554075 9985 connection.cc:512] client connection to 127.4.96.3:33068 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:33068): Connection reset by peer (error 104) W0517 06:44:51.585088 10074 connection.cc:512] client connection to 127.4.96.62:41305 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:41305): Connection reset by peer (error 104) W0517 06:44:51.585186 10199 connection.cc:512] client connection to 127.4.96.62:41305 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:41305): Connection reset by peer (error 104) W0517 06:44:51.585191 10447 connection.cc:512] client connection to 127.4.96.62:41305 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:41305): Connection reset by peer (error 104) W0517 06:44:51.586133 9989 connection.cc:512] client connection to 127.4.96.62:41305 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:41305): Connection reset by peer (error 104) W0517 06:44:51.586828 10070 connection.cc:512] server connection from 127.0.0.1:53983 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:53983): Connection reset by peer (error 104) W0517 06:44:51.586894 10688 connection.cc:512] client connection to 127.4.96.62:41305 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:41305): Connection reset by peer (error 104) I0517 06:44:51.588352 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:41305 --webserver_interface=127.4.96.62 --webserver_port=56915 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:51.703946 10721 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:51.704259 10721 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:51.713151 10721 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:44:51.713270 10721 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:51.713371 10721 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:44:51.713480 10721 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:44:51.727530 10721 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:41305 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=56915 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:51.735208 10721 master_main.cc:80] Initializing master server... I0517 06:44:51.736670 10721 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 144819us I0517 06:44:51.737408 10721 fs_manager.cc:260] Metadata directory not provided I0517 06:44:51.737542 10721 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:51.742830 10721 fs_manager.cc:397] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s I0517 06:44:51.743417 10721 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:51.743801 10721 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:51.750144 10721 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:44:51.750252 10721 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "dba82cf69c2a4538a8be63dd114489da" format_stamp: "Formatted at 2018-05-17 06:44:48 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:51.751348 10721 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0517 06:44:51.892410 10070 consensus_peers.cc:434] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer 15ac5271aef7440e87dea2eaa9321ca2 (127.4.96.3:33068): Couldn't send request to peer 15ac5271aef7440e87dea2eaa9321ca2 for tablet 881bb0d82041492bbd4945aa8c43312a. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.3:33068: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0517 06:44:51.894803 10183 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:41305: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.96.62:41305: connect: Connection refused (error 111) I0517 06:44:51.924526 10721 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:51.926230 10721 master_main.cc:83] Starting Master server... I0517 06:44:51.965941 10721 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:41305 I0517 06:44:51.966780 10721 webserver.cc:173] Starting webserver on 127.4.96.62:56915 I0517 06:44:51.966876 10721 webserver.cc:184] Document root disabled I0517 06:44:51.968047 10721 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:56915/ I0517 06:44:51.968801 10721 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb W0517 06:44:51.971622 10072 consensus_peers.cc:434] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer da1d7cec6f814868a154aa0516cf5098 (127.4.96.5:46584): Couldn't send request to peer da1d7cec6f814868a154aa0516cf5098 for tablet 881bb0d82041492bbd4945aa8c43312a. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:46584: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:51.973773 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 10721 I0517 06:44:51.975898 10140 consensus_peers.cc:456] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer da1d7cec6f814868a154aa0516cf5098 (127.4.96.5:46584): Closing peer: da1d7cec6f814868a154aa0516cf5098 I0517 06:44:51.976080 10140 consensus_peers.cc:456] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer 15ac5271aef7440e87dea2eaa9321ca2 (127.4.96.3:33068): Closing peer: 15ac5271aef7440e87dea2eaa9321ca2 I0517 06:44:51.976505 10140 consensus_queue.cc:204] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } W0517 06:44:51.978588 10072 consensus_peers.cc:434] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer da1d7cec6f814868a154aa0516cf5098 (127.4.96.5:46584): Couldn't send request to peer da1d7cec6f814868a154aa0516cf5098 for tablet 881bb0d82041492bbd4945aa8c43312a. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:46584: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:51.978895 10784 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: Bootstrap starting. I0517 06:44:52.002985 10784 log.cc:520] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: Log is configured to *not* fsync() on all Append() calls I0517 06:44:52.014387 10784 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:52.014892 10784 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: Bootstrap complete. I0517 06:44:52.031906 10784 raft_consensus.cc:303] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } I0517 06:44:52.033409 10784 raft_consensus.cc:604] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba82cf69c2a4538a8be63dd114489da, State: Initialized, Role: FOLLOWER I0517 06:44:52.033874 10784 consensus_queue.cc:227] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } I0517 06:44:52.034245 10784 raft_consensus.cc:343] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:44:52.034412 10784 raft_consensus.cc:435] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:44:52.034555 10784 raft_consensus.cc:2704] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: Snoozing failure detection for 1.827s (starting election) I0517 06:44:52.034698 10784 raft_consensus.cc:2756] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 1 FOLLOWER]: Advancing to term 2 I0517 06:44:52.038784 10784 raft_consensus.cc:457] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } I0517 06:44:52.039188 10784 leader_election.cc:270] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0517 06:44:52.040360 10786 raft_consensus.cc:2704] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: Snoozing failure detection for 1.975s (election complete) I0517 06:44:52.040661 10786 raft_consensus.cc:2511] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 2 FOLLOWER]: Leader election won for term 2 I0517 06:44:52.042227 10786 raft_consensus.cc:571] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [term 2 LEADER]: Becoming Leader. State: Replica: dba82cf69c2a4538a8be63dd114489da, State: Running, Role: LEADER I0517 06:44:52.042583 10786 consensus_queue.cc:204] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } I0517 06:44:52.045620 10787 sys_catalog.cc:335] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dba82cf69c2a4538a8be63dd114489da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } } I0517 06:44:52.046068 10787 sys_catalog.cc:338] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: This master's current role is: LEADER I0517 06:44:52.046857 10784 sys_catalog.cc:335] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "dba82cf69c2a4538a8be63dd114489da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } } I0517 06:44:52.105216 10784 sys_catalog.cc:338] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: This master's current role is: LEADER I0517 06:44:52.106143 10784 sys_catalog.cc:422] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: configured and running, proceeding with master startup. I0517 06:44:52.047111 10788 sys_catalog.cc:335] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: SysCatalogTable state changed. Reason: New leader dba82cf69c2a4538a8be63dd114489da. Latest consensus state: current_term: 2 leader_uuid: "dba82cf69c2a4538a8be63dd114489da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba82cf69c2a4538a8be63dd114489da" member_type: VOTER } } I0517 06:44:52.108059 10788 sys_catalog.cc:338] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da [sys.catalog]: This master's current role is: LEADER I0517 06:44:52.109051 10721 master_main.cc:86] Master server successfully started. I0517 06:44:52.114027 10791 catalog_manager.cc:1045] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: acquiring CA information for follower catalog manager: success I0517 06:44:52.115949 10791 catalog_manager.cc:1073] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0517 06:44:52.116245 10790 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:44:52.119107 10790 catalog_manager.cc:321] Loaded metadata for table TestTable [id=a4d22e57e98a4e1aa50cbd38128cde9c] I0517 06:44:52.121906 10790 catalog_manager.cc:375] Loaded metadata for tablet 881bb0d82041492bbd4945aa8c43312a (table TestTable [id=a4d22e57e98a4e1aa50cbd38128cde9c]) I0517 06:44:52.122506 10790 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:44:52.125180 10790 catalog_manager.cc:1021] Loading token signing keys... I0517 06:44:52.127249 10790 catalog_manager.cc:3973] T 00000000000000000000000000000000 P dba82cf69c2a4538a8be63dd114489da: Loaded TSK: 0 I0517 06:44:52.332973 10751 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "6bdab74f06a741abb63715127aeb2811" instance_seqno: 1526539490251043) as {username='slave'} at 127.4.96.4:43719; Asking this server to re-register. I0517 06:44:52.334313 10556 heartbeater.cc:426] Registering TS with master... I0517 06:44:52.334656 10556 heartbeater.cc:475] Master 127.4.96.62:41305 requested a full tablet report, sending... I0517 06:44:52.335954 10751 ts_manager.cc:81] Registered new tserver with Master: 6bdab74f06a741abb63715127aeb2811 (127.4.96.4:60463) I0517 06:44:52.442725 10751 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" instance_seqno: 1526539489342279) as {username='slave'} at 127.4.96.2:42738; Asking this server to re-register. I0517 06:44:52.443627 10308 heartbeater.cc:426] Registering TS with master... I0517 06:44:52.443884 10308 heartbeater.cc:475] Master 127.4.96.62:41305 requested a full tablet report, sending... I0517 06:44:52.444983 10751 ts_manager.cc:81] Registered new tserver with Master: 28ba6c7ee5464c6db38265b65d3fee0f (127.4.96.2:55398) W0517 06:44:52.566594 10072 consensus_peers.cc:434] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer da1d7cec6f814868a154aa0516cf5098 (127.4.96.5:46584): Couldn't send request to peer da1d7cec6f814868a154aa0516cf5098 for tablet 881bb0d82041492bbd4945aa8c43312a. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:46584: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I0517 06:44:52.905392 10183 heartbeater.cc:346] Connected to a master server at 127.4.96.62:41305 I0517 06:44:52.907127 10751 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" instance_seqno: 1526539488847807) as {username='slave'} at 127.4.96.1:46440; Asking this server to re-register. I0517 06:44:52.908072 10183 heartbeater.cc:426] Registering TS with master... I0517 06:44:52.908303 10183 heartbeater.cc:475] Master 127.4.96.62:41305 requested a full tablet report, sending... I0517 06:44:52.909727 10751 ts_manager.cc:81] Registered new tserver with Master: 4a56ce22b67e4bd795817145c47d2a15 (127.4.96.1:47178) W0517 06:44:52.984942 10072 consensus_peers.cc:434] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer da1d7cec6f814868a154aa0516cf5098 (127.4.96.5:46584): Couldn't send request to peer da1d7cec6f814868a154aa0516cf5098 for tablet 881bb0d82041492bbd4945aa8c43312a. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:46584: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0517 06:44:53.525880 10072 consensus_peers.cc:434] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer da1d7cec6f814868a154aa0516cf5098 (127.4.96.5:46584): Couldn't send request to peer da1d7cec6f814868a154aa0516cf5098 for tablet 881bb0d82041492bbd4945aa8c43312a. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:46584: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0517 06:44:54.151027 10072 consensus_peers.cc:434] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer da1d7cec6f814868a154aa0516cf5098 (127.4.96.5:46584): Couldn't send request to peer da1d7cec6f814868a154aa0516cf5098 for tablet 881bb0d82041492bbd4945aa8c43312a. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:46584: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. I0517 06:44:54.302855 10140 tablet_service.cc:1138] Received LeaderStepDown RPC: tablet_id: "881bb0d82041492bbd4945aa8c43312a" dest_uuid: "4a56ce22b67e4bd795817145c47d2a15" from {username='slave'} at 127.0.0.1:53995 I0517 06:44:54.303179 10140 raft_consensus.cc:530] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 1 LEADER]: Received request to step down I0517 06:44:54.303299 10140 raft_consensus.cc:604] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a56ce22b67e4bd795817145c47d2a15, State: Running, Role: LEADER I0517 06:44:54.303630 10140 consensus_queue.cc:227] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } I0517 06:44:54.303984 10140 consensus_peers.cc:456] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer da1d7cec6f814868a154aa0516cf5098 (127.4.96.5:46584): Closing peer: da1d7cec6f814868a154aa0516cf5098 I0517 06:44:54.304193 10140 raft_consensus.cc:2704] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Snoozing failure detection for 3.000s (explicit stepdown request) I0517 06:44:55.325491 4480 kudu-admin-test.cc:881] Change Config Op timed out, Sending a Replace config command when change config op is pending on the leader. W0517 06:44:55.439855 10821 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:44:55.646780 10140 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "4a56ce22b67e4bd795817145c47d2a15" tablet_id: "881bb0d82041492bbd4945aa8c43312a" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" } } from {username='slave'} at 127.0.0.1:53997 W0517 06:44:55.647236 10140 raft_consensus.cc:1891] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 1 FOLLOWER]: Replica has a pending config, but the new config will be unsafely changed anyway. Currently pending config on the node: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } W0517 06:44:55.647765 10140 raft_consensus.cc:1990] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 1 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 33068 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } unsafe_config_change: true I0517 06:44:55.648351 10140 raft_consensus.cc:2756] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:44:55.651772 10140 raft_consensus.cc:2823] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 2 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } }; New pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } unsafe_config_change: true I0517 06:44:57.153085 10840 raft_consensus.cc:435] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0517 06:44:57.153311 10840 raft_consensus.cc:2704] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Snoozing failure detection for 1.608s (starting election) I0517 06:44:57.153486 10840 raft_consensus.cc:457] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } unsafe_config_change: true I0517 06:44:57.153879 10840 leader_election.cc:270] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0517 06:44:57.154624 10840 raft_consensus.cc:2704] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Snoozing failure detection for 2.003s (election complete) I0517 06:44:57.154788 10840 raft_consensus.cc:2511] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0517 06:44:57.154924 10840 raft_consensus.cc:435] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0517 06:44:57.155045 10840 raft_consensus.cc:2704] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Snoozing failure detection for 1.696s (starting election) I0517 06:44:57.155164 10840 raft_consensus.cc:2756] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 2 FOLLOWER]: Advancing to term 3 I0517 06:44:57.157862 10840 raft_consensus.cc:457] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } unsafe_config_change: true I0517 06:44:57.158241 10840 leader_election.cc:270] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0517 06:44:57.158663 10841 raft_consensus.cc:2704] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Snoozing failure detection for 1.878s (election complete) I0517 06:44:57.158846 10841 raft_consensus.cc:2511] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 3 FOLLOWER]: Leader election won for term 3 I0517 06:44:57.159085 10841 raft_consensus.cc:571] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 3 LEADER]: Becoming Leader. State: Replica: 4a56ce22b67e4bd795817145c47d2a15, State: Running, Role: LEADER I0517 06:44:57.159377 10841 consensus_queue.cc:204] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.3, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } unsafe_config_change: true I0517 06:44:57.161252 10841 raft_consensus.cc:2658] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 3 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 15ac5271aef7440e87dea2eaa9321ca2 (127.4.96.3) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } } I0517 06:44:57.162385 10751 catalog_manager.cc:3686] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 reported cstate change: term changed from 1 to 3, now has a pending config: VOTER 4a56ce22b67e4bd795817145c47d2a15 (127.4.96.1). New cstate: current_term: 3 leader_uuid: "4a56ce22b67e4bd795817145c47d2a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ac5271aef7440e87dea2eaa9321ca2" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 33068 } } peers { permanent_uuid: "da1d7cec6f814868a154aa0516cf5098" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46584 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } unsafe_config_change: true } W0517 06:44:57.164758 10841 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:53993 (request call id 33) took 5189ms (client timeout 2000). W0517 06:44:57.165385 10841 rpcz_store.cc:255] Trace: 0517 06:44:51.975078 (+ 0us) service_pool.cc:162] Inserting onto call queue 0517 06:44:51.975173 (+ 95us) service_pool.cc:221] Handling call 0517 06:44:51.977434 (+ 2261us) log.cc:582] Serialized 364 byte log entry 0517 06:44:57.164712 (+5187278us) inbound_call.cc:162] Queueing success response Metrics: {"dns_us":35,"raft.queue_time_us":77,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":195} I0517 06:44:57.165714 10841 raft_consensus.cc:2658] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 3 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 15ac5271aef7440e87dea2eaa9321ca2 (127.4.96.3) evicted, VOTER da1d7cec6f814868a154aa0516cf5098 (127.4.96.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } unsafe_config_change: true } I0517 06:44:57.180061 10751 catalog_manager.cc:3686] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 reported cstate change: config changed from index -1 to 3, VOTER 15ac5271aef7440e87dea2eaa9321ca2 (127.4.96.3) evicted, VOTER da1d7cec6f814868a154aa0516cf5098 (127.4.96.5) evicted, no longer has a pending config: VOTER 4a56ce22b67e4bd795817145c47d2a15 (127.4.96.1). New cstate: current_term: 3 leader_uuid: "4a56ce22b67e4bd795817145c47d2a15" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:44:57.184056 10751 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2 with a delay of 26 ms (attempt = 1) W0517 06:44:57.184253 10751 catalog_manager.cc:3845] Failed to send 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2: Not found: failed to reset TS proxy: Could not find TS for UUID 15ac5271aef7440e87dea2eaa9321ca2 I0517 06:44:57.184479 10751 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098 with a delay of 21 ms (attempt = 1) W0517 06:44:57.184630 10751 catalog_manager.cc:3845] Failed to send 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098: Not found: failed to reset TS proxy: Could not find TS for UUID da1d7cec6f814868a154aa0516cf5098 I0517 06:44:57.185092 10751 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 881bb0d82041492bbd4945aa8c43312a (attempt 1) I0517 06:44:57.195791 10140 consensus_queue.cc:204] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:57.197245 10840 raft_consensus.cc:2658] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, NON_VOTER 6bdab74f06a741abb63715127aeb2811 (127.4.96.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:57.202723 10731 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 881bb0d82041492bbd4945aa8c43312a with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0517 06:44:57.203821 10751 catalog_manager.cc:3686] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 reported cstate change: config changed from index 3 to 5, NON_VOTER 6bdab74f06a741abb63715127aeb2811 (127.4.96.4) added. New cstate: current_term: 3 leader_uuid: "4a56ce22b67e4bd795817145c47d2a15" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0517 06:44:57.206288 10731 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098 with a delay of 50 ms (attempt = 2) W0517 06:44:57.206472 10731 catalog_manager.cc:2810] Async tablet task 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098 failed: Not found: failed to reset TS proxy: Could not find TS for UUID da1d7cec6f814868a154aa0516cf5098 I0517 06:44:57.207548 10751 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 881bb0d82041492bbd4945aa8c43312a (attempt 1) I0517 06:44:57.209197 10140 consensus_peers.cc:456] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer 6bdab74f06a741abb63715127aeb2811 (127.4.96.4:60463): Closing peer: 6bdab74f06a741abb63715127aeb2811 I0517 06:44:57.209532 10140 consensus_queue.cc:204] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: true } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:57.210675 10729 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2 with a delay of 43 ms (attempt = 2) W0517 06:44:57.210891 10729 catalog_manager.cc:2810] Async tablet task 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 15ac5271aef7440e87dea2eaa9321ca2 I0517 06:44:57.211339 10841 raft_consensus.cc:2658] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER 28ba6c7ee5464c6db38265b65d3fee0f (127.4.96.2) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: true } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: true } } unsafe_config_change: true } W0517 06:44:57.212852 10070 consensus_peers.cc:434] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer 6bdab74f06a741abb63715127aeb2811 (127.4.96.4:60463): Couldn't send request to peer 6bdab74f06a741abb63715127aeb2811 for tablet 881bb0d82041492bbd4945aa8c43312a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 881bb0d82041492bbd4945aa8c43312a. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:57.214998 10731 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 881bb0d82041492bbd4945aa8c43312a with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0517 06:44:57.224794 10751 catalog_manager.cc:3686] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 reported cstate change: config changed from index 5 to 6, NON_VOTER 28ba6c7ee5464c6db38265b65d3fee0f (127.4.96.2) added. New cstate: current_term: 3 leader_uuid: "4a56ce22b67e4bd795817145c47d2a15" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } W0517 06:44:57.232316 10071 consensus_peers.cc:434] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer 28ba6c7ee5464c6db38265b65d3fee0f (127.4.96.2:55398): Couldn't send request to peer 28ba6c7ee5464c6db38265b65d3fee0f for tablet 881bb0d82041492bbd4945aa8c43312a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 881bb0d82041492bbd4945aa8c43312a. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:44:57.254528 10729 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2 with a delay of 106 ms (attempt = 3) W0517 06:44:57.254896 10729 catalog_manager.cc:2810] Async tablet task 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 15ac5271aef7440e87dea2eaa9321ca2 I0517 06:44:57.257566 10731 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098 with a delay of 96 ms (attempt = 3) W0517 06:44:57.257731 10731 catalog_manager.cc:2810] Async tablet task 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098 failed: Not found: failed to reset TS proxy: Could not find TS for UUID da1d7cec6f814868a154aa0516cf5098 I0517 06:44:57.354315 10731 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098 with a delay of 174 ms (attempt = 4) W0517 06:44:57.354475 10731 catalog_manager.cc:2810] Async tablet task 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098 failed: Not found: failed to reset TS proxy: Could not find TS for UUID da1d7cec6f814868a154aa0516cf5098 I0517 06:44:57.361498 10729 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2 with a delay of 157 ms (attempt = 4) W0517 06:44:57.361671 10729 catalog_manager.cc:2810] Async tablet task 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 15ac5271aef7440e87dea2eaa9321ca2 I0517 06:44:57.519554 10729 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2 with a delay of 277 ms (attempt = 5) W0517 06:44:57.519717 10729 catalog_manager.cc:2810] Async tablet task 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 15ac5271aef7440e87dea2eaa9321ca2 I0517 06:44:57.529245 10731 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098 with a delay of 276 ms (attempt = 5) W0517 06:44:57.529429 10731 catalog_manager.cc:2810] Async tablet task 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098 failed: Not found: failed to reset TS proxy: Could not find TS for UUID da1d7cec6f814868a154aa0516cf5098 I0517 06:44:57.720234 10853 ts_tablet_manager.cc:669] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: Initiating tablet copy from peer 4a56ce22b67e4bd795817145c47d2a15 (127.4.96.1:47178) I0517 06:44:57.720521 10853 tablet_copy_client.cc:239] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.1:47178 I0517 06:44:57.730064 10160 tablet_copy_service.cc:133] P 4a56ce22b67e4bd795817145c47d2a15: Received BeginTabletCopySession request for tablet 881bb0d82041492bbd4945aa8c43312a from peer 28ba6c7ee5464c6db38265b65d3fee0f ({username='slave'} at 127.4.96.2:39795) I0517 06:44:57.730300 10160 tablet_copy_service.cc:154] P 4a56ce22b67e4bd795817145c47d2a15: Beginning new tablet copy session on tablet 881bb0d82041492bbd4945aa8c43312a from peer 28ba6c7ee5464c6db38265b65d3fee0f at {username='slave'} at 127.4.96.2:39795: session id = 28ba6c7ee5464c6db38265b65d3fee0f-881bb0d82041492bbd4945aa8c43312a I0517 06:44:57.733901 10160 tablet_copy_source_session.cc:202] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:44:57.736495 10853 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 881bb0d82041492bbd4945aa8c43312a. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:57.747068 10853 ts_tablet_manager.cc:1173] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:44:57.747445 10853 tablet_copy_client.cc:565] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: tablet copy: Starting download of 0 data blocks... I0517 06:44:57.747674 10853 tablet_copy_client.cc:528] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: tablet copy: Starting download of 1 WAL segments... I0517 06:44:57.749590 10853 tablet_copy_client.cc:404] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:44:57.752224 10853 ts_tablet_manager.cc:1028] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: Bootstrapping tablet I0517 06:44:57.752543 10853 tablet_bootstrap.cc:436] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: Bootstrap starting. I0517 06:44:57.757627 10853 log.cc:520] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: Log is configured to *not* fsync() on all Append() calls I0517 06:44:57.763964 10853 tablet_bootstrap.cc:436] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:57.764484 10853 tablet_bootstrap.cc:436] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: Bootstrap complete. I0517 06:44:57.764843 10853 ts_tablet_manager.cc:1045] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: Time spent bootstrapping tablet: real 0.012s user 0.012s sys 0.004s I0517 06:44:57.774245 10853 raft_consensus.cc:303] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: true } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:57.774835 10853 raft_consensus.cc:604] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 28ba6c7ee5464c6db38265b65d3fee0f, State: Initialized, Role: LEARNER I0517 06:44:57.775166 10853 consensus_queue.cc:227] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: true } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:57.777380 10853 ts_tablet_manager.cc:1073] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s I0517 06:44:57.778409 10160 tablet_copy_service.cc:323] P 4a56ce22b67e4bd795817145c47d2a15: Request end of tablet copy session 28ba6c7ee5464c6db38265b65d3fee0f-881bb0d82041492bbd4945aa8c43312a received from {username='slave'} at 127.4.96.2:39795 I0517 06:44:57.778583 10160 tablet_copy_service.cc:411] P 4a56ce22b67e4bd795817145c47d2a15: Ending tablet copy session 28ba6c7ee5464c6db38265b65d3fee0f-881bb0d82041492bbd4945aa8c43312a on tablet 881bb0d82041492bbd4945aa8c43312a with peer 28ba6c7ee5464c6db38265b65d3fee0f I0517 06:44:57.797910 10729 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2 with a delay of 546 ms (attempt = 6) W0517 06:44:57.798095 10729 catalog_manager.cc:2810] Async tablet task 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 15ac5271aef7440e87dea2eaa9321ca2 I0517 06:44:57.806210 10731 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098 with a delay of 534 ms (attempt = 6) W0517 06:44:57.806361 10731 catalog_manager.cc:2810] Async tablet task 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098 failed: Not found: failed to reset TS proxy: Could not find TS for UUID da1d7cec6f814868a154aa0516cf5098 I0517 06:44:57.834252 10858 ts_tablet_manager.cc:669] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: Initiating tablet copy from peer 4a56ce22b67e4bd795817145c47d2a15 (127.4.96.1:47178) I0517 06:44:57.834555 10858 tablet_copy_client.cc:239] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.1:47178 I0517 06:44:57.842890 10160 tablet_copy_service.cc:133] P 4a56ce22b67e4bd795817145c47d2a15: Received BeginTabletCopySession request for tablet 881bb0d82041492bbd4945aa8c43312a from peer 6bdab74f06a741abb63715127aeb2811 ({username='slave'} at 127.4.96.4:45673) I0517 06:44:57.843089 10160 tablet_copy_service.cc:154] P 4a56ce22b67e4bd795817145c47d2a15: Beginning new tablet copy session on tablet 881bb0d82041492bbd4945aa8c43312a from peer 6bdab74f06a741abb63715127aeb2811 at {username='slave'} at 127.4.96.4:45673: session id = 6bdab74f06a741abb63715127aeb2811-881bb0d82041492bbd4945aa8c43312a I0517 06:44:57.846001 10160 tablet_copy_source_session.cc:202] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:44:57.848455 10858 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 881bb0d82041492bbd4945aa8c43312a. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:57.859290 10858 ts_tablet_manager.cc:1173] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:44:57.859638 10858 tablet_copy_client.cc:565] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: tablet copy: Starting download of 0 data blocks... I0517 06:44:57.859920 10858 tablet_copy_client.cc:528] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: tablet copy: Starting download of 1 WAL segments... I0517 06:44:57.861793 10858 tablet_copy_client.cc:404] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:44:57.864568 10858 ts_tablet_manager.cc:1028] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: Bootstrapping tablet I0517 06:44:57.864894 10858 tablet_bootstrap.cc:436] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: Bootstrap starting. I0517 06:44:57.870123 10858 log.cc:520] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: Log is configured to *not* fsync() on all Append() calls I0517 06:44:57.876076 10858 tablet_bootstrap.cc:436] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:44:57.876507 10858 tablet_bootstrap.cc:436] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: Bootstrap complete. I0517 06:44:57.876792 10858 ts_tablet_manager.cc:1045] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.004s I0517 06:44:57.886041 10858 raft_consensus.cc:303] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: true } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:57.886574 10858 raft_consensus.cc:604] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 6bdab74f06a741abb63715127aeb2811, State: Initialized, Role: LEARNER I0517 06:44:57.886873 10858 consensus_queue.cc:227] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: true } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:57.888856 10858 ts_tablet_manager.cc:1073] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: Time spent starting tablet: real 0.012s user 0.016s sys 0.000s I0517 06:44:57.889797 10160 tablet_copy_service.cc:323] P 4a56ce22b67e4bd795817145c47d2a15: Request end of tablet copy session 6bdab74f06a741abb63715127aeb2811-881bb0d82041492bbd4945aa8c43312a received from {username='slave'} at 127.4.96.4:45673 I0517 06:44:57.889983 10160 tablet_copy_service.cc:411] P 4a56ce22b67e4bd795817145c47d2a15: Ending tablet copy session 6bdab74f06a741abb63715127aeb2811-881bb0d82041492bbd4945aa8c43312a on tablet 881bb0d82041492bbd4945aa8c43312a with peer 6bdab74f06a741abb63715127aeb2811 W0517 06:44:57.894731 10309 tablet.cc:1678] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:44:58.038805 10557 tablet.cc:1678] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:44:58.205461 10265 raft_consensus.cc:1002] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0517 06:44:58.340786 10731 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098 with a delay of 1051 ms (attempt = 7) W0517 06:44:58.340965 10731 catalog_manager.cc:2810] Async tablet task 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=da1d7cec6f814868a154aa0516cf5098 failed: Not found: failed to reset TS proxy: Could not find TS for UUID da1d7cec6f814868a154aa0516cf5098 I0517 06:44:58.344605 10729 catalog_manager.cc:2791] Scheduling retry of 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2 with a delay of 1058 ms (attempt = 7) W0517 06:44:58.344768 10729 catalog_manager.cc:2810] Async tablet task 881bb0d82041492bbd4945aa8c43312a Delete Tablet RPC for TS=15ac5271aef7440e87dea2eaa9321ca2 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 15ac5271aef7440e87dea2eaa9321ca2 I0517 06:44:58.354276 10513 raft_consensus.cc:1002] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0517 06:44:58.739754 10863 raft_consensus.cc:866] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: attempting to promote NON_VOTER 6bdab74f06a741abb63715127aeb2811 to VOTER I0517 06:44:58.740532 10863 consensus_peers.cc:456] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer 28ba6c7ee5464c6db38265b65d3fee0f (127.4.96.2:55398): Closing peer: 28ba6c7ee5464c6db38265b65d3fee0f I0517 06:44:58.740689 10863 consensus_peers.cc:456] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer 6bdab74f06a741abb63715127aeb2811 (127.4.96.4:60463): Closing peer: 6bdab74f06a741abb63715127aeb2811 I0517 06:44:58.741118 10863 consensus_queue.cc:204] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: false } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: true } } unsafe_config_change: true I0517 06:44:58.744398 10513 raft_consensus.cc:1057] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811 [term 3 LEARNER]: Refusing update from remote peer 4a56ce22b67e4bd795817145c47d2a15: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:44:58.744621 10265 raft_consensus.cc:1057] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f [term 3 LEARNER]: Refusing update from remote peer 4a56ce22b67e4bd795817145c47d2a15: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:44:58.745389 10862 consensus_queue.cc:963] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:44:58.745820 10841 consensus_queue.cc:963] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:44:58.750814 10862 raft_consensus.cc:829] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 3 LEADER]: attempt to promote peer 28ba6c7ee5464c6db38265b65d3fee0f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing. I0517 06:44:58.751077 10841 raft_consensus.cc:2658] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 6bdab74f06a741abb63715127aeb2811 (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: false } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:58.752046 10513 raft_consensus.cc:2658] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 6bdab74f06a741abb63715127aeb2811 (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: false } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:58.756937 10265 raft_consensus.cc:2658] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f [term 3 LEARNER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 6bdab74f06a741abb63715127aeb2811 (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: false } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:58.757905 10751 catalog_manager.cc:3686] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811 reported cstate change: config changed from index 6 to 7, 6bdab74f06a741abb63715127aeb2811 (127.4.96.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "4a56ce22b67e4bd795817145c47d2a15" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: false } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: NON_VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: true } } unsafe_config_change: true } I0517 06:44:58.765000 10863 raft_consensus.cc:866] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15: attempting to promote NON_VOTER 28ba6c7ee5464c6db38265b65d3fee0f to VOTER I0517 06:44:58.765673 10863 consensus_peers.cc:456] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer 28ba6c7ee5464c6db38265b65d3fee0f (127.4.96.2:55398): Closing peer: 28ba6c7ee5464c6db38265b65d3fee0f I0517 06:44:58.765854 10863 consensus_peers.cc:456] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 -> Peer 6bdab74f06a741abb63715127aeb2811 (127.4.96.4:60463): Closing peer: 6bdab74f06a741abb63715127aeb2811 I0517 06:44:58.766435 10863 consensus_queue.cc:204] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: false } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: false } } unsafe_config_change: true I0517 06:44:58.779062 10265 raft_consensus.cc:1057] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f [term 3 LEARNER]: Refusing update from remote peer 4a56ce22b67e4bd795817145c47d2a15: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0517 06:44:58.779537 10512 raft_consensus.cc:1057] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811 [term 3 FOLLOWER]: Refusing update from remote peer 4a56ce22b67e4bd795817145c47d2a15: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0517 06:44:58.779882 10841 consensus_queue.cc:963] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0517 06:44:58.780642 10841 consensus_queue.cc:963] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0517 06:44:58.783427 10862 raft_consensus.cc:2658] T 881bb0d82041492bbd4945aa8c43312a P 4a56ce22b67e4bd795817145c47d2a15 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 28ba6c7ee5464c6db38265b65d3fee0f (127.4.96.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: false } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:58.784622 10265 raft_consensus.cc:2658] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 28ba6c7ee5464c6db38265b65d3fee0f (127.4.96.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: false } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:58.785846 10512 raft_consensus.cc:2658] T 881bb0d82041492bbd4945aa8c43312a P 6bdab74f06a741abb63715127aeb2811 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 28ba6c7ee5464c6db38265b65d3fee0f (127.4.96.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: false } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:58.791607 10750 catalog_manager.cc:3686] T 881bb0d82041492bbd4945aa8c43312a P 28ba6c7ee5464c6db38265b65d3fee0f reported cstate change: config changed from index 7 to 8, 28ba6c7ee5464c6db38265b65d3fee0f (127.4.96.2) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "4a56ce22b67e4bd795817145c47d2a15" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 47178 } } peers { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 60463 } attrs { promote: false } } peers { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 55398 } attrs { promote: false } } unsafe_config_change: true } I0517 06:44:58.888890 4480 kudu-admin-test.cc:907] Waiting for Master to see new config... I0517 06:44:58.898305 4480 kudu-admin-test.cc:912] Tablet locations: tablet_id: "881bb0d82041492bbd4945aa8c43312a" replicas { ts_info { permanent_uuid: "4a56ce22b67e4bd795817145c47d2a15" rpc_addresses { host: "127.4.96.1" port: 47178 } } role: LEADER } replicas { ts_info { permanent_uuid: "6bdab74f06a741abb63715127aeb2811" rpc_addresses { host: "127.4.96.4" port: 60463 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "28ba6c7ee5464c6db38265b65d3fee0f" rpc_addresses { host: "127.4.96.2" port: 55398 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0517 06:44:58.899600 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 10052 W0517 06:44:58.920748 10730 connection.cc:512] server connection from 127.4.96.1:46440 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:46440): Connection reset by peer (error 104) W0517 06:44:58.920785 10731 connection.cc:512] client connection to 127.4.96.1:47178 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:47178): Connection reset by peer (error 104) W0517 06:44:58.920847 9989 connection.cc:512] client connection to 127.4.96.1:47178 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:47178): Connection reset by peer (error 104) W0517 06:44:58.921869 10447 connection.cc:512] server connection from 127.4.96.1:38961 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:38961): Connection reset by peer (error 104) W0517 06:44:58.921972 10197 connection.cc:512] client connection to 127.4.96.1:47178 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:47178): Connection reset by peer (error 104) W0517 06:44:58.922736 9987 connection.cc:512] client connection to 127.4.96.1:47178 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:47178): Connection reset by peer (error 104) W0517 06:44:58.923563 10445 connection.cc:512] client connection to 127.4.96.1:47178 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:47178): Connection reset by peer (error 104) W0517 06:44:58.923563 10195 connection.cc:512] server connection from 127.4.96.1:58253 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:58253): Connection reset by peer (error 104) I0517 06:44:58.924582 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 10187 W0517 06:44:58.944424 10735 connection.cc:512] server connection from 127.4.96.2:42738 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:42738): Connection reset by peer (error 104) I0517 06:44:58.945338 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 10435 W0517 06:44:58.958081 9989 connection.cc:512] client connection to 127.4.96.2:55398 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:55398): Connection reset by peer (error 104) W0517 06:44:58.974493 10735 connection.cc:512] server connection from 127.4.96.4:43719 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:43719): Connection reset by peer (error 104) W0517 06:44:58.974648 9989 connection.cc:512] client connection to 127.4.96.4:60463 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:60463): Connection reset by peer (error 104) W0517 06:44:58.975375 9985 connection.cc:512] client connection to 127.4.96.4:60463 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:60463): Connection reset by peer (error 104) I0517 06:44:58.976210 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 10721 W0517 06:44:59.003384 9989 connection.cc:512] client connection to 127.4.96.62:41305 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:41305): Connection reset by peer (error 104) [ OK ] AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig (11094 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL I0517 06:44:59.017314 4480 test_util.cc:200] Using random seed: 747836494 I0517 06:44:59.020228 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:44:59.020336 4480 ts_itest-base.cc:114] -------------- I0517 06:44:59.020437 4480 ts_itest-base.cc:115] 5 tablet servers I0517 06:44:59.020522 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:44:59.020613 4480 ts_itest-base.cc:117] -------------- I0517 06:44:59.022879 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:60575 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:59.135076 10883 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:59.135412 10883 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:44:59.144428 10883 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:44:59.144562 10883 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:59.144672 10883 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:44:59.144798 10883 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:44:59.158639 10883 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:60575 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:59.166275 10883 master_main.cc:80] Initializing master server... I0517 06:44:59.167793 10883 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 148819us I0517 06:44:59.168536 10883 fs_manager.cc:260] Metadata directory not provided I0517 06:44:59.168675 10883 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:59.168887 10883 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:44:59.169003 10883 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:59.176151 10883 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "d0a0e44bf01a44398b3a85b01e20edf9" format_stamp: "Formatted at 2018-05-17 06:44:59 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:59.176789 10883 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "d0a0e44bf01a44398b3a85b01e20edf9" format_stamp: "Formatted at 2018-05-17 06:44:59 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:59.186255 10883 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:44:59.187335 10883 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:59.187736 10883 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:59.194077 10883 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:44:59.194190 10883 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "d0a0e44bf01a44398b3a85b01e20edf9" format_stamp: "Formatted at 2018-05-17 06:44:59 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:59.194397 10883 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:59.355824 10883 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:59.357533 10883 master_main.cc:83] Starting Master server... I0517 06:44:59.396795 10883 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:60575 I0517 06:44:59.397619 10883 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:44:59.397712 10883 webserver.cc:184] Document root disabled I0517 06:44:59.398906 10883 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:55278/ I0517 06:44:59.399597 10883 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:44:59.402101 10944 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:44:59.405246 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 10883 I0517 06:44:59.406635 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60575 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:59.418315 10944 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: Bootstrap starting. I0517 06:44:59.420577 10944 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: No blocks or log segments found. Creating new log. I0517 06:44:59.421747 10944 log.cc:520] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: Log is configured to *not* fsync() on all Append() calls I0517 06:44:59.424064 10944 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: No bootstrap required, opened a new log I0517 06:44:59.435717 10944 raft_consensus.cc:303] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } I0517 06:44:59.436025 10944 raft_consensus.cc:329] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:44:59.436151 10944 raft_consensus.cc:604] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a0e44bf01a44398b3a85b01e20edf9, State: Initialized, Role: FOLLOWER I0517 06:44:59.436574 10944 consensus_queue.cc:227] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } I0517 06:44:59.436928 10944 raft_consensus.cc:343] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:44:59.437065 10944 raft_consensus.cc:435] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:44:59.437208 10944 raft_consensus.cc:2704] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: Snoozing failure detection for 1.579s (starting election) I0517 06:44:59.437346 10944 raft_consensus.cc:2756] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:44:59.439757 10944 raft_consensus.cc:457] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } I0517 06:44:59.440173 10944 leader_election.cc:270] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:44:59.441699 10947 raft_consensus.cc:2704] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: Snoozing failure detection for 1.822s (election complete) I0517 06:44:59.442031 10947 raft_consensus.cc:2511] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:44:59.450407 10947 raft_consensus.cc:571] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 1 LEADER]: Becoming Leader. State: Replica: d0a0e44bf01a44398b3a85b01e20edf9, State: Running, Role: LEADER I0517 06:44:59.450824 10947 consensus_queue.cc:204] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } I0517 06:44:59.453902 10944 sys_catalog.cc:335] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } } I0517 06:44:59.454476 10944 sys_catalog.cc:338] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: This master's current role is: LEADER I0517 06:44:59.456843 10944 sys_catalog.cc:422] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:44:59.458518 10883 master_main.cc:86] Master server successfully started. I0517 06:44:59.458046 10948 sys_catalog.cc:335] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } } I0517 06:44:59.517169 10948 sys_catalog.cc:338] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: This master's current role is: LEADER WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:44:59.518172 10945 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:44:59.518508 10945 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:44:59.458492 10951 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:44:59.458788 10949 sys_catalog.cc:335] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0a0e44bf01a44398b3a85b01e20edf9. Latest consensus state: current_term: 1 leader_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } } I0517 06:44:59.521720 10949 sys_catalog.cc:338] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: This master's current role is: LEADER I0517 06:44:59.524122 10951 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... W0517 06:44:59.530565 10945 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:44:59.531035 10945 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:44:59.568866 10945 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60575 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:44:59.580262 10951 catalog_manager.cc:899] Generated new certificate authority record I0517 06:44:59.581224 10945 tablet_server_main.cc:85] Initializing tablet server... I0517 06:44:59.582592 10951 catalog_manager.cc:1021] Loading token signing keys... I0517 06:44:59.583084 10945 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 148819us I0517 06:44:59.583541 10945 fs_manager.cc:260] Metadata directory not provided I0517 06:44:59.583665 10945 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:44:59.583885 10945 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:44:59.584007 10945 server_base.cc:433] Attempting to create new FS layout instead I0517 06:44:59.591737 10945 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" format_stamp: "Formatted at 2018-05-17 06:44:59 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:59.592571 10951 catalog_manager.cc:3962] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: Generated new TSK 0 I0517 06:44:59.592742 10945 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" format_stamp: "Formatted at 2018-05-17 06:44:59 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:59.603049 10945 fs_manager.cc:495] Time spent creating directory manager: real 0.010s user 0.008s sys 0.004s I0517 06:44:59.604169 10945 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:44:59.604578 10945 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:44:59.611147 10945 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.004s sys 0.004s I0517 06:44:59.611258 10945 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" format_stamp: "Formatted at 2018-05-17 06:44:59 on dist-test-slave-dist-test-slave-26mv" I0517 06:44:59.611470 10945 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:44:59.830623 10945 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:44:59.833431 10945 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:44:59.834575 10945 tablet_server_main.cc:90] Starting tablet server... I0517 06:44:59.936681 10945 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:49563 I0517 06:44:59.937584 10945 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:44:59.937690 10945 webserver.cc:184] Document root disabled I0517 06:44:59.939107 10945 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:41322/ I0517 06:44:59.939803 10945 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:44:59.943835 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 10945 I0517 06:44:59.945063 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60575 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:44:59.950170 10945 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:44:59.977583 11076 heartbeater.cc:346] Connected to a master server at 127.4.96.62:60575 I0517 06:44:59.977890 11076 heartbeater.cc:426] Registering TS with master... I0517 06:44:59.978480 11076 heartbeater.cc:475] Master 127.4.96.62:60575 requested a full tablet report, sending... I0517 06:44:59.980518 10911 ts_manager.cc:81] Registered new tserver with Master: 6c3dcb1ce66b4b3a8e14a7305a1af218 (127.4.96.1:49563) I0517 06:44:59.983165 10911 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:51070 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:00.050817 11080 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:00.051153 11080 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:00.059116 11080 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:00.059453 11080 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:45:00.072227 11080 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60575 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:00.080255 11080 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:00.082046 11080 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 149319us I0517 06:45:00.082479 11080 fs_manager.cc:260] Metadata directory not provided I0517 06:45:00.082607 11080 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:00.082816 11080 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:45:00.082934 11080 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:00.090397 11080 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" format_stamp: "Formatted at 2018-05-17 06:45:00 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:00.091054 11080 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" format_stamp: "Formatted at 2018-05-17 06:45:00 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:00.100885 11080 fs_manager.cc:495] Time spent creating directory manager: real 0.010s user 0.008s sys 0.004s I0517 06:45:00.102025 11080 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:00.102430 11080 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:00.108954 11080 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:45:00.109067 11080 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" format_stamp: "Formatted at 2018-05-17 06:45:00 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:00.109282 11080 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:00.319697 11080 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:00.322484 11080 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:00.323704 11080 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:00.443089 11080 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:33771 I0517 06:45:00.444334 11080 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:45:00.444478 11080 webserver.cc:184] Document root disabled I0517 06:45:00.445967 11080 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:41737/ I0517 06:45:00.446859 11080 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:45:00.451961 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 11080 I0517 06:45:00.453109 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60575 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:00.470330 11080 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:00.495772 11201 heartbeater.cc:346] Connected to a master server at 127.4.96.62:60575 I0517 06:45:00.496032 11201 heartbeater.cc:426] Registering TS with master... I0517 06:45:00.496587 11201 heartbeater.cc:475] Master 127.4.96.62:60575 requested a full tablet report, sending... I0517 06:45:00.498391 10911 ts_manager.cc:81] Registered new tserver with Master: 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2:33771) I0517 06:45:00.500107 10911 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:52482 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:00.558465 11204 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:00.558787 11204 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:00.566766 11204 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:00.567078 11204 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:45:00.579905 11204 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60575 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:00.588107 11204 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:00.589876 11204 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 149319us I0517 06:45:00.590323 11204 fs_manager.cc:260] Metadata directory not provided I0517 06:45:00.590432 11204 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:00.590616 11204 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:45:00.590728 11204 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:00.598609 11204 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "a0e9f9fef615453a89e52dc146476b94" format_stamp: "Formatted at 2018-05-17 06:45:00 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:00.599251 11204 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "a0e9f9fef615453a89e52dc146476b94" format_stamp: "Formatted at 2018-05-17 06:45:00 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:00.608898 11204 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:45:00.610144 11204 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:00.610534 11204 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:00.617069 11204 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:45:00.617174 11204 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "a0e9f9fef615453a89e52dc146476b94" format_stamp: "Formatted at 2018-05-17 06:45:00 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:00.617358 11204 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:00.816303 11204 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:00.818986 11204 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:00.820106 11204 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:00.905779 11204 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:49197 I0517 06:45:00.906687 11204 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:45:00.906795 11204 webserver.cc:184] Document root disabled I0517 06:45:00.908044 11204 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:47792/ I0517 06:45:00.908805 11204 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:45:00.913048 11204 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:00.918359 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 11204 I0517 06:45:00.919271 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60575 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:00.929020 11325 heartbeater.cc:346] Connected to a master server at 127.4.96.62:60575 I0517 06:45:00.929260 11325 heartbeater.cc:426] Registering TS with master... I0517 06:45:00.929695 11325 heartbeater.cc:475] Master 127.4.96.62:60575 requested a full tablet report, sending... I0517 06:45:00.931098 10911 ts_manager.cc:81] Registered new tserver with Master: a0e9f9fef615453a89e52dc146476b94 (127.4.96.3:49197) I0517 06:45:00.932194 10911 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:42632 I0517 06:45:00.986357 11076 heartbeater.cc:467] Master 127.4.96.62:60575 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:01.025202 11328 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:01.025563 11328 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:01.033680 11328 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:01.034029 11328 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:45:01.046505 11328 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60575 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:01.054823 11328 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:01.056632 11328 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 149819us I0517 06:45:01.057093 11328 fs_manager.cc:260] Metadata directory not provided I0517 06:45:01.057202 11328 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:01.057395 11328 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:45:01.057508 11328 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:01.065093 11328 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "23a5a54ae9484026bfe44987841a7b20" format_stamp: "Formatted at 2018-05-17 06:45:01 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:01.065752 11328 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "23a5a54ae9484026bfe44987841a7b20" format_stamp: "Formatted at 2018-05-17 06:45:01 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:01.075271 11328 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:45:01.076434 11328 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:01.076839 11328 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:01.083191 11328 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.004s I0517 06:45:01.083302 11328 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "23a5a54ae9484026bfe44987841a7b20" format_stamp: "Formatted at 2018-05-17 06:45:01 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:01.083492 11328 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:01.234624 11328 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:01.236994 11328 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:01.237917 11328 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:01.360788 11328 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:43655 I0517 06:45:01.361714 11328 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:45:01.361814 11328 webserver.cc:184] Document root disabled I0517 06:45:01.363359 11328 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:51695/ I0517 06:45:01.364043 11328 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:45:01.364946 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 11328 I0517 06:45:01.366155 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.4.96.5:0 --local_ip_for_outbound_sockets=127.4.96.5 --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60575 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:01.395345 11328 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:01.420768 11449 heartbeater.cc:346] Connected to a master server at 127.4.96.62:60575 I0517 06:45:01.421057 11449 heartbeater.cc:426] Registering TS with master... I0517 06:45:01.421682 11449 heartbeater.cc:475] Master 127.4.96.62:60575 requested a full tablet report, sending... I0517 06:45:01.423317 10911 ts_manager.cc:81] Registered new tserver with Master: 23a5a54ae9484026bfe44987841a7b20 (127.4.96.4:43655) I0517 06:45:01.424666 10911 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:48852 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:01.500775 11450 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:01.501150 11450 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:45:01.503535 11201 heartbeater.cc:467] Master 127.4.96.62:60575 was elected leader, sending a full tablet report... W0517 06:45:01.510072 11450 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:01.510402 11450 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.5 I0517 06:45:01.523229 11450 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60575 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.5 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:01.531489 11450 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:01.533267 11450 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 149819us I0517 06:45:01.533694 11450 fs_manager.cc:260] Metadata directory not provided I0517 06:45:01.533828 11450 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:01.534055 11450 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0517 06:45:01.534178 11450 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:01.541821 11450 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "63382e9ca0f7430ba027db6b03202ab2" format_stamp: "Formatted at 2018-05-17 06:45:01 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:01.542498 11450 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "63382e9ca0f7430ba027db6b03202ab2" format_stamp: "Formatted at 2018-05-17 06:45:01 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:01.552176 11450 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:45:01.553339 11450 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:01.553772 11450 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:01.560305 11450 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:45:01.560422 11450 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal uuid: "63382e9ca0f7430ba027db6b03202ab2" format_stamp: "Formatted at 2018-05-17 06:45:01 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:01.560644 11450 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:01.726219 11450 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:01.728540 11450 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:01.729462 11450 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:01.814990 11450 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.5:46864 I0517 06:45:01.815878 11450 webserver.cc:173] Starting webserver on 127.4.96.5:0 I0517 06:45:01.815985 11450 webserver.cc:184] Document root disabled I0517 06:45:01.817384 11450 webserver.cc:311] Webserver started. Bound to: http://127.4.96.5:39641/ I0517 06:45:01.818120 11450 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb I0517 06:45:01.822947 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 11450 I0517 06:45:01.823155 11450 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:01.842392 11573 heartbeater.cc:346] Connected to a master server at 127.4.96.62:60575 I0517 06:45:01.842669 11573 heartbeater.cc:426] Registering TS with master... I0517 06:45:01.843202 11573 heartbeater.cc:475] Master 127.4.96.62:60575 requested a full tablet report, sending... I0517 06:45:01.844461 10911 ts_manager.cc:81] Registered new tserver with Master: 63382e9ca0f7430ba027db6b03202ab2 (127.4.96.5:46864) I0517 06:45:01.845713 10911 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.5:43273 I0517 06:45:01.846495 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:45:01.864132 10911 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34046: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:45:01.914064 11013 tablet_service.cc:759] Processing CreateTablet for tablet 1ab1a102dd9c49aba0780da8d8e19b23 (table=TestTable [id=626dc1e294184dfb95f3d8cd061a5804]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:01.914790 11013 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1ab1a102dd9c49aba0780da8d8e19b23. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:01.920164 11262 tablet_service.cc:759] Processing CreateTablet for tablet 1ab1a102dd9c49aba0780da8d8e19b23 (table=TestTable [id=626dc1e294184dfb95f3d8cd061a5804]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:01.920804 11138 tablet_service.cc:759] Processing CreateTablet for tablet 1ab1a102dd9c49aba0780da8d8e19b23 (table=TestTable [id=626dc1e294184dfb95f3d8cd061a5804]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:01.921033 11262 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1ab1a102dd9c49aba0780da8d8e19b23. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:01.921429 11138 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1ab1a102dd9c49aba0780da8d8e19b23. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:01.929369 11013 ts_tablet_manager.cc:1173] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:01.933192 11593 ts_tablet_manager.cc:1028] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Bootstrapping tablet I0517 06:45:01.933656 11593 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Bootstrap starting. I0517 06:45:01.934063 11138 ts_tablet_manager.cc:1173] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:01.936604 11262 ts_tablet_manager.cc:1173] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:01.936841 11593 tablet_bootstrap.cc:587] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: No blocks or log segments found. Creating new log. I0517 06:45:01.937914 11594 ts_tablet_manager.cc:1028] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24: Bootstrapping tablet I0517 06:45:01.938426 11594 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24: Bootstrap starting. I0517 06:45:01.939406 11593 log.cc:520] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Log is configured to *not* fsync() on all Append() calls I0517 06:45:01.941323 11594 tablet_bootstrap.cc:587] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24: No blocks or log segments found. Creating new log. I0517 06:45:01.942865 11594 log.cc:520] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24: Log is configured to *not* fsync() on all Append() calls I0517 06:45:01.945580 11593 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: No bootstrap required, opened a new log I0517 06:45:01.945946 11593 ts_tablet_manager.cc:1045] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s I0517 06:45:01.946254 11594 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24: No bootstrap required, opened a new log I0517 06:45:01.946631 11594 ts_tablet_manager.cc:1045] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s I0517 06:45:01.961382 11594 raft_consensus.cc:303] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } I0517 06:45:01.962225 11594 raft_consensus.cc:329] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:01.962440 11594 raft_consensus.cc:604] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432b3fcdbb4c4dc0aa368a7f02647c24, State: Initialized, Role: FOLLOWER I0517 06:45:01.962427 11593 raft_consensus.cc:303] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } I0517 06:45:01.963168 11593 raft_consensus.cc:329] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:01.963366 11593 raft_consensus.cc:604] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3dcb1ce66b4b3a8e14a7305a1af218, State: Initialized, Role: FOLLOWER I0517 06:45:01.963114 11594 consensus_queue.cc:227] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } I0517 06:45:01.963966 11593 consensus_queue.cc:227] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } I0517 06:45:01.972767 11593 ts_tablet_manager.cc:1073] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Time spent starting tablet: real 0.027s user 0.016s sys 0.004s I0517 06:45:01.974611 11594 ts_tablet_manager.cc:1073] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24: Time spent starting tablet: real 0.028s user 0.024s sys 0.000s I0517 06:45:01.934914 11325 heartbeater.cc:467] Master 127.4.96.62:60575 was elected leader, sending a full tablet report... I0517 06:45:02.017190 11595 ts_tablet_manager.cc:1028] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94: Bootstrapping tablet I0517 06:45:02.017751 11595 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94: Bootstrap starting. I0517 06:45:02.020547 11595 tablet_bootstrap.cc:587] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94: No blocks or log segments found. Creating new log. I0517 06:45:02.021806 11595 log.cc:520] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94: Log is configured to *not* fsync() on all Append() calls I0517 06:45:02.024047 11595 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94: No bootstrap required, opened a new log I0517 06:45:02.024320 11595 ts_tablet_manager.cc:1045] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94: Time spent bootstrapping tablet: real 0.007s user 0.008s sys 0.000s I0517 06:45:02.034420 11595 raft_consensus.cc:303] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } I0517 06:45:02.034953 11595 raft_consensus.cc:329] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:02.035086 11595 raft_consensus.cc:604] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e9f9fef615453a89e52dc146476b94, State: Initialized, Role: FOLLOWER I0517 06:45:02.035715 11595 consensus_queue.cc:227] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } I0517 06:45:02.037691 11595 ts_tablet_manager.cc:1073] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s I0517 06:45:02.089619 11599 raft_consensus.cc:435] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:02.089867 11599 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Snoozing failure detection for 1.758s (starting election) I0517 06:45:02.090092 11599 raft_consensus.cc:457] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } I0517 06:45:02.091033 11599 leader_election.cc:243] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a0e9f9fef615453a89e52dc146476b94 (127.4.96.3:49197) I0517 06:45:02.091475 11599 leader_election.cc:243] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2:33771) I0517 06:45:02.102632 11282 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1ab1a102dd9c49aba0780da8d8e19b23" candidate_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e9f9fef615453a89e52dc146476b94" is_pre_election: true I0517 06:45:02.103091 11282 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94: Snoozing failure detection for 1.904s (vote granted) I0517 06:45:02.103291 11282 raft_consensus.cc:2219] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c3dcb1ce66b4b3a8e14a7305a1af218 in term 0. I0517 06:45:02.104344 10963 leader_election.cc:379] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a0e9f9fef615453a89e52dc146476b94 (127.4.96.3:49197) I0517 06:45:02.104565 10963 leader_election.cc:270] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:45:02.105064 11599 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Snoozing failure detection for 1.937s (election complete) I0517 06:45:02.105249 11599 raft_consensus.cc:2511] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:45:02.105131 11158 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1ab1a102dd9c49aba0780da8d8e19b23" candidate_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" is_pre_election: true I0517 06:45:02.105404 11599 raft_consensus.cc:435] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:45:02.105566 11599 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Snoozing failure detection for 1.567s (starting election) I0517 06:45:02.105578 11158 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24: Snoozing failure detection for 2.248s (vote granted) I0517 06:45:02.105720 11599 raft_consensus.cc:2756] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:02.105772 11158 raft_consensus.cc:2219] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c3dcb1ce66b4b3a8e14a7305a1af218 in term 0. I0517 06:45:02.109835 11599 raft_consensus.cc:457] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } I0517 06:45:02.110708 11599 leader_election.cc:243] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [CANDIDATE]: Term 1 election: Requesting vote from peer a0e9f9fef615453a89e52dc146476b94 (127.4.96.3:49197) I0517 06:45:02.111089 11599 leader_election.cc:243] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [CANDIDATE]: Term 1 election: Requesting vote from peer 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2:33771) I0517 06:45:02.111765 11282 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1ab1a102dd9c49aba0780da8d8e19b23" candidate_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e9f9fef615453a89e52dc146476b94" I0517 06:45:02.112066 11282 raft_consensus.cc:2756] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:02.112270 11282 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94: Snoozing failure detection for 1.907s (vote granted) I0517 06:45:02.115190 11282 raft_consensus.cc:2219] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c3dcb1ce66b4b3a8e14a7305a1af218 in term 1. I0517 06:45:02.106637 10965 leader_election.cc:379] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2:33771) W0517 06:45:02.161581 10965 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.96.2:33771, user_credentials={real_user=slave}} blocked reactor thread for 55038.9us W0517 06:45:02.162778 11326 tablet.cc:1678] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:02.162714 11158 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1ab1a102dd9c49aba0780da8d8e19b23" candidate_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" I0517 06:45:02.163139 11158 raft_consensus.cc:2756] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:02.163414 11158 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24: Snoozing failure detection for 1.514s (vote granted) I0517 06:45:02.115872 10963 leader_election.cc:379] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [CANDIDATE]: Term 1 election: Vote granted by peer a0e9f9fef615453a89e52dc146476b94 (127.4.96.3:49197) I0517 06:45:02.164377 10963 leader_election.cc:270] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:02.164713 11599 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Snoozing failure detection for 1.650s (election complete) W0517 06:45:02.164672 10963 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.96.3:49197, user_credentials={real_user=slave}} blocked reactor thread for 48820.1us I0517 06:45:02.164965 11599 raft_consensus.cc:2511] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:02.166232 11599 raft_consensus.cc:571] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 1 LEADER]: Becoming Leader. State: Replica: 6c3dcb1ce66b4b3a8e14a7305a1af218, State: Running, Role: LEADER I0517 06:45:02.166792 11158 raft_consensus.cc:2219] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c3dcb1ce66b4b3a8e14a7305a1af218 in term 1. I0517 06:45:02.166618 11599 consensus_queue.cc:204] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } I0517 06:45:02.167589 10965 leader_election.cc:379] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [CANDIDATE]: Term 1 election: Vote granted by peer 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2:33771) I0517 06:45:02.171442 10910 catalog_manager.cc:3686] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 reported cstate change: term changed from 0 to 1, leader changed from to 6c3dcb1ce66b4b3a8e14a7305a1af218 (127.4.96.1). New cstate: current_term: 1 leader_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } health_report { overall_health: UNKNOWN } } } W0517 06:45:02.201980 11077 tablet.cc:1678] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:02.203290 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:45:02.205274 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6c3dcb1ce66b4b3a8e14a7305a1af218 to finish bootstrapping I0517 06:45:02.214658 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 432b3fcdbb4c4dc0aa368a7f02647c24 to finish bootstrapping W0517 06:45:02.221750 11202 tablet.cc:1678] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:02.223636 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a0e9f9fef615453a89e52dc146476b94 to finish bootstrapping I0517 06:45:02.232266 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 23a5a54ae9484026bfe44987841a7b20 to finish bootstrapping I0517 06:45:02.242810 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 63382e9ca0f7430ba027db6b03202ab2 to finish bootstrapping I0517 06:45:02.252874 4480 kudu-admin-test.cc:943] Waiting for Master to see the current replicas... I0517 06:45:02.254859 4480 kudu-admin-test.cc:950] Tablet locations: tablet_id: "1ab1a102dd9c49aba0780da8d8e19b23" replicas { ts_info { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" rpc_addresses { host: "127.4.96.1" port: 49563 } } role: LEADER } replicas { ts_info { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" rpc_addresses { host: "127.4.96.3" port: 49197 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" rpc_addresses { host: "127.4.96.2" port: 33771 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0517 06:45:02.427119 11449 heartbeater.cc:467] Master 127.4.96.62:60575 was elected leader, sending a full tablet report... I0517 06:45:02.712115 11282 raft_consensus.cc:1057] T 1ab1a102dd9c49aba0780da8d8e19b23 P a0e9f9fef615453a89e52dc146476b94 [term 1 FOLLOWER]: Refusing update from remote peer 6c3dcb1ce66b4b3a8e14a7305a1af218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:45:02.713181 11609 consensus_queue.cc:963] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:02.722314 11158 raft_consensus.cc:1057] T 1ab1a102dd9c49aba0780da8d8e19b23 P 432b3fcdbb4c4dc0aa368a7f02647c24 [term 1 FOLLOWER]: Refusing update from remote peer 6c3dcb1ce66b4b3a8e14a7305a1af218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:45:02.724345 11609 consensus_queue.cc:963] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:02.742439 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 11080 W0517 06:45:02.760748 10965 connection.cc:512] client connection to 127.4.96.2:33771 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:33771): Connection reset by peer (error 104) W0517 06:45:02.760763 10893 connection.cc:512] client connection to 127.4.96.2:33771 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:33771): Connection reset by peer (error 104) W0517 06:45:02.761672 10892 connection.cc:512] server connection from 127.4.96.2:52482 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:52482): Connection reset by peer (error 104) W0517 06:45:02.761759 10882 connection.cc:512] client connection to 127.4.96.2:33771 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:33771): Connection reset by peer (error 104) W0517 06:45:02.762624 10880 connection.cc:512] client connection to 127.4.96.2:33771 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:33771): Connection reset by peer (error 104) I0517 06:45:02.762765 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 11204 W0517 06:45:02.779925 10891 connection.cc:512] client connection to 127.4.96.3:49197 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:49197): Connection reset by peer (error 104) W0517 06:45:02.779961 10963 connection.cc:512] client connection to 127.4.96.3:49197 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:49197): Connection reset by peer (error 104) W0517 06:45:02.779995 10882 connection.cc:512] client connection to 127.4.96.3:49197 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:49197): Connection reset by peer (error 104) W0517 06:45:02.780664 10891 connection.cc:512] server connection from 127.4.96.3:42632 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:42632): Connection reset by peer (error 104) W0517 06:45:02.780860 10878 connection.cc:512] client connection to 127.4.96.3:49197 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:49197): Connection reset by peer (error 104) I0517 06:45:02.782402 11033 consensus_peers.cc:456] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2:33771): Closing peer: 432b3fcdbb4c4dc0aa368a7f02647c24 I0517 06:45:02.782564 11033 consensus_peers.cc:456] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer a0e9f9fef615453a89e52dc146476b94 (127.4.96.3:49197): Closing peer: a0e9f9fef615453a89e52dc146476b94 I0517 06:45:02.782954 11033 consensus_queue.cc:204] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } W0517 06:45:02.786263 10965 consensus_peers.cc:434] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2:33771): Couldn't send request to peer 432b3fcdbb4c4dc0aa368a7f02647c24 for tablet 1ab1a102dd9c49aba0780da8d8e19b23. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.2:33771: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0517 06:45:02.848220 11573 heartbeater.cc:467] Master 127.4.96.62:60575 was elected leader, sending a full tablet report... W0517 06:45:03.307034 10965 consensus_peers.cc:434] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2:33771): Couldn't send request to peer 432b3fcdbb4c4dc0aa368a7f02647c24 for tablet 1ab1a102dd9c49aba0780da8d8e19b23. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.2:33771: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0517 06:45:03.728747 10965 consensus_peers.cc:434] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2:33771): Couldn't send request to peer 432b3fcdbb4c4dc0aa368a7f02647c24 for tablet 1ab1a102dd9c49aba0780da8d8e19b23. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.2:33771: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0517 06:45:04.149147 10965 consensus_peers.cc:434] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2:33771): Couldn't send request to peer 432b3fcdbb4c4dc0aa368a7f02647c24 for tablet 1ab1a102dd9c49aba0780da8d8e19b23. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.2:33771: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0517 06:45:04.591812 10965 consensus_peers.cc:434] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2:33771): Couldn't send request to peer 432b3fcdbb4c4dc0aa368a7f02647c24 for tablet 1ab1a102dd9c49aba0780da8d8e19b23. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.2:33771: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. I0517 06:45:04.782272 4480 kudu-admin-test.cc:971] Change Config Op timed out, Sending a Replace config command when change config op is pending on the leader. W0517 06:45:04.896486 11619 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:04.998867 10965 consensus_peers.cc:434] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2:33771): Couldn't send request to peer 432b3fcdbb4c4dc0aa368a7f02647c24 for tablet 1ab1a102dd9c49aba0780da8d8e19b23. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.2:33771: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. I0517 06:45:05.090134 11033 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" tablet_id: "1ab1a102dd9c49aba0780da8d8e19b23" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" } } from {username='slave'} at 127.0.0.1:49266 W0517 06:45:05.090569 11033 raft_consensus.cc:1891] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 1 LEADER]: Replica has a pending config, but the new config will be unsafely changed anyway. Currently pending config on the node: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } W0517 06:45:05.091084 11033 raft_consensus.cc:1990] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 1 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } unsafe_config_change: true I0517 06:45:05.091711 11033 raft_consensus.cc:2751] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 1 LEADER]: Stepping down as leader of term 1 I0517 06:45:05.091847 11033 raft_consensus.cc:604] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c3dcb1ce66b4b3a8e14a7305a1af218, State: Running, Role: LEADER I0517 06:45:05.092159 11033 consensus_queue.cc:227] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } I0517 06:45:05.092548 11033 consensus_peers.cc:456] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2:33771): Closing peer: 432b3fcdbb4c4dc0aa368a7f02647c24 I0517 06:45:05.092763 11033 raft_consensus.cc:2756] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:45:05.095769 11033 raft_consensus.cc:2823] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } }; New pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } unsafe_config_change: true W0517 06:45:05.159847 10968 debug-util.cc:397] Leaking SignalData structure 0x7b080002b080 after lost signal to thread 10954 W0517 06:45:06.111094 11073 generic_service.cc:126] {username='slave'} at 127.0.0.1:49257 forcing change of non-runtime-safe flag fault_crash_before_cmeta_flush I0517 06:45:06.111335 11073 generic_service.cc:146] {username='slave'} at 127.0.0.1:49257 changed flags via RPC: fault_crash_before_cmeta_flush from '0' to '1.0' I0517 06:45:06.112465 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 10883 W0517 06:45:06.135809 10965 connection.cc:512] client connection to 127.4.96.62:60575 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:60575): Connection reset by peer (error 104) W0517 06:45:06.135952 11338 connection.cc:512] client connection to 127.4.96.62:60575 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:60575): Connection reset by peer (error 104) W0517 06:45:06.137125 11462 connection.cc:512] client connection to 127.4.96.62:60575 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:60575): Connection reset by peer (error 104) W0517 06:45:06.137096 10882 connection.cc:512] client connection to 127.4.96.62:60575 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:60575): Connection reset by peer (error 104) W0517 06:45:06.138661 10964 connection.cc:512] server connection from 127.0.0.1:49254 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:49254): Connection reset by peer (error 104) W0517 06:45:06.138712 11580 connection.cc:512] client connection to 127.4.96.62:60575 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:60575): Connection reset by peer (error 104) I0517 06:45:06.140233 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:60575 --webserver_interface=127.4.96.62 --webserver_port=55278 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:06.253890 11635 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:06.254232 11635 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:06.263100 11635 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:45:06.263236 11635 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:06.263348 11635 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:45:06.263476 11635 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:45:06.277604 11635 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:60575 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=55278 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:06.285452 11635 master_main.cc:80] Initializing master server... I0517 06:45:06.287068 11635 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 152319us I0517 06:45:06.287899 11635 fs_manager.cc:260] Metadata directory not provided I0517 06:45:06.288050 11635 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:06.293277 11635 fs_manager.cc:397] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s I0517 06:45:06.293884 11635 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:06.294311 11635 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:06.300755 11635 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:45:06.300873 11635 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "d0a0e44bf01a44398b3a85b01e20edf9" format_stamp: "Formatted at 2018-05-17 06:44:59 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:06.302012 11635 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0517 06:45:06.438925 11449 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:60575: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.96.62:60575: connect: Connection refused (error 111) I0517 06:45:06.449894 11635 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:06.451689 11635 master_main.cc:83] Starting Master server... I0517 06:45:06.490592 11635 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:60575 I0517 06:45:06.491443 11635 webserver.cc:173] Starting webserver on 127.4.96.62:55278 I0517 06:45:06.491544 11635 webserver.cc:184] Document root disabled I0517 06:45:06.492923 11635 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:55278/ I0517 06:45:06.493669 11635 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:45:06.502472 11697 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: Bootstrap starting. I0517 06:45:06.503070 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 11635 I0517 06:45:06.526365 11697 log.cc:520] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: Log is configured to *not* fsync() on all Append() calls I0517 06:45:06.537225 11697 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:45:06.537762 11697 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: Bootstrap complete. I0517 06:45:06.554922 11697 raft_consensus.cc:303] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } I0517 06:45:06.556460 11697 raft_consensus.cc:604] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a0e44bf01a44398b3a85b01e20edf9, State: Initialized, Role: FOLLOWER I0517 06:45:06.557003 11697 consensus_queue.cc:227] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } I0517 06:45:06.557449 11697 raft_consensus.cc:343] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:45:06.557618 11697 raft_consensus.cc:435] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:45:06.557796 11697 raft_consensus.cc:2704] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: Snoozing failure detection for 2.093s (starting election) I0517 06:45:06.557973 11697 raft_consensus.cc:2756] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:45:06.562213 11697 raft_consensus.cc:457] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } I0517 06:45:06.562636 11697 leader_election.cc:270] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0517 06:45:06.564054 11699 raft_consensus.cc:2704] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: Snoozing failure detection for 1.740s (election complete) I0517 06:45:06.564270 11699 raft_consensus.cc:2511] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 2 FOLLOWER]: Leader election won for term 2 I0517 06:45:06.565542 11699 raft_consensus.cc:571] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [term 2 LEADER]: Becoming Leader. State: Replica: d0a0e44bf01a44398b3a85b01e20edf9, State: Running, Role: LEADER I0517 06:45:06.565969 11699 consensus_queue.cc:204] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } I0517 06:45:06.568720 11699 sys_catalog.cc:335] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0a0e44bf01a44398b3a85b01e20edf9. Latest consensus state: current_term: 2 leader_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } } I0517 06:45:06.569697 11699 sys_catalog.cc:338] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: This master's current role is: LEADER I0517 06:45:06.610322 11704 raft_consensus.cc:435] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0517 06:45:06.610640 11704 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Snoozing failure detection for 2.206s (starting election) I0517 06:45:06.610868 11704 raft_consensus.cc:457] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } unsafe_config_change: true I0517 06:45:06.611380 11704 leader_election.cc:270] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0517 06:45:06.615761 11705 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Snoozing failure detection for 1.557s (election complete) I0517 06:45:06.616019 11705 raft_consensus.cc:2511] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0517 06:45:06.616220 11705 raft_consensus.cc:435] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0517 06:45:06.616420 11705 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Snoozing failure detection for 1.592s (starting election) I0517 06:45:06.616611 11705 raft_consensus.cc:2756] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Advancing to term 3 W0517 06:45:06.616830 11705 fault_injection.cc:41] FAULT INJECTION ENABLED! W0517 06:45:06.616951 11705 fault_injection.cc:42] THIS SERVER MAY CRASH! E0517 06:45:06.617082 11705 fault_injection.cc:57] Injecting fault: FLAGS_fault_crash_before_cmeta_flush (process will exit) I0517 06:45:06.569532 11701 sys_catalog.cc:335] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } } I0517 06:45:06.660282 11701 sys_catalog.cc:338] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: This master's current role is: LEADER I0517 06:45:06.662314 11697 sys_catalog.cc:335] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a0e44bf01a44398b3a85b01e20edf9" member_type: VOTER } } I0517 06:45:06.662681 11697 sys_catalog.cc:338] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: This master's current role is: LEADER I0517 06:45:06.663511 11697 sys_catalog.cc:422] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:45:06.663702 11703 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:45:06.665065 11635 master_main.cc:86] Master server successfully started. I0517 06:45:06.668225 11703 catalog_manager.cc:321] Loaded metadata for table TestTable [id=626dc1e294184dfb95f3d8cd061a5804] I0517 06:45:06.670205 11703 catalog_manager.cc:375] Loaded metadata for tablet 1ab1a102dd9c49aba0780da8d8e19b23 (table TestTable [id=626dc1e294184dfb95f3d8cd061a5804]) I0517 06:45:06.670608 11703 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:45:06.674139 11703 catalog_manager.cc:1021] Loading token signing keys... I0517 06:45:06.675536 11703 catalog_manager.cc:3973] T 00000000000000000000000000000000 P d0a0e44bf01a44398b3a85b01e20edf9: Loaded TSK: 0 I0517 06:45:06.868607 11664 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" instance_seqno: 1526539501797044) as {username='slave'} at 127.4.96.5:55411; Asking this server to re-register. I0517 06:45:06.870007 11573 heartbeater.cc:426] Registering TS with master... I0517 06:45:06.870267 11573 heartbeater.cc:475] Master 127.4.96.62:60575 requested a full tablet report, sending... I0517 06:45:06.871632 11664 ts_manager.cc:81] Registered new tserver with Master: 63382e9ca0f7430ba027db6b03202ab2 (127.4.96.5:46864) I0517 06:45:07.112012 11664 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" instance_seqno: 1526539499915208) as {username='slave'} at 127.4.96.1:37271; Asking this server to re-register. I0517 06:45:07.112992 11076 heartbeater.cc:426] Registering TS with master... I0517 06:45:07.113260 11076 heartbeater.cc:475] Master 127.4.96.62:60575 requested a full tablet report, sending... I0517 06:45:07.455441 11449 heartbeater.cc:346] Connected to a master server at 127.4.96.62:60575 I0517 06:45:07.457070 11664 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" instance_seqno: 1526539501337546) as {username='slave'} at 127.4.96.4:57213; Asking this server to re-register. I0517 06:45:07.458240 11449 heartbeater.cc:426] Registering TS with master... I0517 06:45:07.458534 11449 heartbeater.cc:475] Master 127.4.96.62:60575 requested a full tablet report, sending... I0517 06:45:07.459928 11664 ts_manager.cc:81] Registered new tserver with Master: 23a5a54ae9484026bfe44987841a7b20 (127.4.96.4:43655) W0517 06:45:07.640022 11644 connection.cc:512] server connection from 127.4.96.1:37271 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:37271): Connection reset by peer (error 104) W0517 06:45:07.641392 10882 connection.cc:512] client connection to 127.4.96.1:49563 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:49563): Connection reset by peer (error 104) W0517 06:45:07.642957 10879 connection.cc:512] client connection to 127.4.96.1:49563 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:49563): Connection reset by peer (error 104) I0517 06:45:07.705739 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:49563 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_port=41322 --webserver_interface=127.4.96.1 --tserver_master_addrs=127.4.96.62:60575 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:07.810781 11714 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:07.811106 11714 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:07.819303 11714 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:07.819633 11714 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:45:07.831984 11714 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:49563 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=41322 --tserver_master_addrs=127.4.96.62:60575 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:07.840265 11714 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:07.842072 11714 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 152819us I0517 06:45:07.842489 11714 fs_manager.cc:260] Metadata directory not provided I0517 06:45:07.842618 11714 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:07.847757 11714 fs_manager.cc:397] Time spent opening directory manager: real 0.003s user 0.000s sys 0.008s I0517 06:45:07.848362 11714 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:07.848790 11714 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:07.855365 11714 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:45:07.855481 11714 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" format_stamp: "Formatted at 2018-05-17 06:44:59 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:07.856572 11714 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:08.027864 11714 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:08.030306 11714 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0517 06:45:08.030436 11714 ts_tablet_manager.cc:1005] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Loading tablet metadata I0517 06:45:08.034469 11714 ts_tablet_manager.cc:356] Loaded tablet metadata (1 live tablets) I0517 06:45:08.037292 11714 ts_tablet_manager.cc:1173] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:08.040644 11731 ts_tablet_manager.cc:1028] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Bootstrapping tablet I0517 06:45:08.041097 11731 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Bootstrap starting. I0517 06:45:08.041007 11714 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:08.069152 11731 log.cc:520] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Log is configured to *not* fsync() on all Append() calls I0517 06:45:08.074358 11731 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates I0517 06:45:08.074925 11731 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Bootstrap complete. I0517 06:45:08.075922 11731 ts_tablet_manager.cc:1045] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.004s I0517 06:45:08.099223 11731 raft_consensus.cc:303] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Replica starting. Triggering 2 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } I0517 06:45:08.106608 11731 raft_consensus.cc:2823] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } }; New pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } unsafe_config_change: true I0517 06:45:08.107465 11731 raft_consensus.cc:604] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3dcb1ce66b4b3a8e14a7305a1af218, State: Initialized, Role: FOLLOWER I0517 06:45:08.107987 11731 consensus_queue.cc:227] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } unsafe_config_change: true I0517 06:45:08.110648 11731 ts_tablet_manager.cc:1073] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Time spent starting tablet: real 0.035s user 0.036s sys 0.000s I0517 06:45:08.167352 11714 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:49563 I0517 06:45:08.168243 11714 webserver.cc:173] Starting webserver on 127.4.96.1:41322 I0517 06:45:08.168345 11714 webserver.cc:184] Document root disabled I0517 06:45:08.169550 11714 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:41322/ I0517 06:45:08.170253 11714 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:45:08.173808 11714 tablet_server_main.cc:93] Tablet server successfully started. W0517 06:45:08.178846 11839 tablet.cc:1678] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:08.180452 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 11714 I0517 06:45:08.190732 11838 heartbeater.cc:346] Connected to a master server at 127.4.96.62:60575 I0517 06:45:08.191005 11838 heartbeater.cc:426] Registering TS with master... I0517 06:45:08.191633 11838 heartbeater.cc:475] Master 127.4.96.62:60575 requested a full tablet report, sending... I0517 06:45:08.193534 11664 ts_manager.cc:81] Registered new tserver with Master: 6c3dcb1ce66b4b3a8e14a7305a1af218 (127.4.96.1:49563) I0517 06:45:08.196059 11664 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:54248 I0517 06:45:09.198812 11838 heartbeater.cc:467] Master 127.4.96.62:60575 was elected leader, sending a full tablet report... I0517 06:45:09.746322 11845 raft_consensus.cc:435] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:09.746539 11845 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Snoozing failure detection for 2.091s (starting election) I0517 06:45:09.746762 11845 raft_consensus.cc:457] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } unsafe_config_change: true I0517 06:45:09.747289 11845 leader_election.cc:270] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0517 06:45:09.748378 11845 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Snoozing failure detection for 2.228s (election complete) I0517 06:45:09.748550 11845 raft_consensus.cc:2511] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0517 06:45:09.748708 11845 raft_consensus.cc:435] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:45:09.748836 11845 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Snoozing failure detection for 2.019s (starting election) I0517 06:45:09.748966 11845 raft_consensus.cc:2756] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 2 FOLLOWER]: Advancing to term 3 I0517 06:45:09.752307 11845 raft_consensus.cc:457] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } unsafe_config_change: true I0517 06:45:09.752676 11845 leader_election.cc:270] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0517 06:45:09.753098 11846 raft_consensus.cc:2704] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Snoozing failure detection for 2.176s (election complete) I0517 06:45:09.753275 11846 raft_consensus.cc:2511] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 3 FOLLOWER]: Leader election won for term 3 I0517 06:45:09.753489 11846 raft_consensus.cc:571] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 3 LEADER]: Becoming Leader. State: Replica: 6c3dcb1ce66b4b3a8e14a7305a1af218, State: Running, Role: LEADER I0517 06:45:09.753872 11846 consensus_queue.cc:204] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } unsafe_config_change: true I0517 06:45:09.756402 11846 raft_consensus.cc:2658] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 3 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a0e9f9fef615453a89e52dc146476b94 (127.4.96.3) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } } I0517 06:45:09.758122 11664 catalog_manager.cc:3686] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 reported cstate change: term changed from 1 to 3, now has a pending config: VOTER 6c3dcb1ce66b4b3a8e14a7305a1af218 (127.4.96.1). New cstate: current_term: 3 leader_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e9f9fef615453a89e52dc146476b94" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49197 } } peers { permanent_uuid: "432b3fcdbb4c4dc0aa368a7f02647c24" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33771 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } unsafe_config_change: true } I0517 06:45:09.761076 11846 raft_consensus.cc:2658] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 3 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2) evicted, VOTER a0e9f9fef615453a89e52dc146476b94 (127.4.96.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } unsafe_config_change: true } I0517 06:45:09.777396 11664 catalog_manager.cc:3686] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 reported cstate change: config changed from index -1 to 3, VOTER 432b3fcdbb4c4dc0aa368a7f02647c24 (127.4.96.2) evicted, VOTER a0e9f9fef615453a89e52dc146476b94 (127.4.96.3) evicted, no longer has a pending config: VOTER 6c3dcb1ce66b4b3a8e14a7305a1af218 (127.4.96.1). New cstate: current_term: 3 leader_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:45:09.780746 11664 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94 with a delay of 26 ms (attempt = 1) W0517 06:45:09.780930 11664 catalog_manager.cc:3845] Failed to send 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94: Not found: failed to reset TS proxy: Could not find TS for UUID a0e9f9fef615453a89e52dc146476b94 I0517 06:45:09.781138 11664 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24 with a delay of 21 ms (attempt = 1) W0517 06:45:09.781270 11664 catalog_manager.cc:3845] Failed to send 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24: Not found: failed to reset TS proxy: Could not find TS for UUID 432b3fcdbb4c4dc0aa368a7f02647c24 I0517 06:45:09.781647 11664 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1ab1a102dd9c49aba0780da8d8e19b23 (attempt 1) I0517 06:45:09.791815 11795 consensus_queue.cc:204] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } } unsafe_config_change: true I0517 06:45:09.793459 11845 raft_consensus.cc:2658] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, NON_VOTER 23a5a54ae9484026bfe44987841a7b20 (127.4.96.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } } unsafe_config_change: true } I0517 06:45:09.798898 11644 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1ab1a102dd9c49aba0780da8d8e19b23 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0517 06:45:09.800546 11664 catalog_manager.cc:3686] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 reported cstate change: config changed from index 3 to 5, NON_VOTER 23a5a54ae9484026bfe44987841a7b20 (127.4.96.4) added. New cstate: current_term: 3 leader_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0517 06:45:09.802649 11645 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24 with a delay of 50 ms (attempt = 2) W0517 06:45:09.802886 11645 catalog_manager.cc:2810] Async tablet task 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 432b3fcdbb4c4dc0aa368a7f02647c24 I0517 06:45:09.804522 11664 catalog_manager.cc:3325] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1ab1a102dd9c49aba0780da8d8e19b23 (attempt 1) I0517 06:45:09.806082 11795 consensus_peers.cc:456] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 23a5a54ae9484026bfe44987841a7b20 (127.4.96.4:43655): Closing peer: 23a5a54ae9484026bfe44987841a7b20 I0517 06:45:09.806399 11795 consensus_queue.cc:204] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: true } } unsafe_config_change: true I0517 06:45:09.807344 11643 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94 with a delay of 43 ms (attempt = 2) W0517 06:45:09.807549 11643 catalog_manager.cc:2810] Async tablet task 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a0e9f9fef615453a89e52dc146476b94 I0517 06:45:09.808370 11845 raft_consensus.cc:2658] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER 63382e9ca0f7430ba027db6b03202ab2 (127.4.96.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: true } } unsafe_config_change: true } W0517 06:45:09.812455 11724 consensus_peers.cc:434] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 23a5a54ae9484026bfe44987841a7b20 (127.4.96.4:43655): Couldn't send request to peer 23a5a54ae9484026bfe44987841a7b20 for tablet 1ab1a102dd9c49aba0780da8d8e19b23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ab1a102dd9c49aba0780da8d8e19b23. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:45:09.813859 11644 catalog_manager.cc:3240] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1ab1a102dd9c49aba0780da8d8e19b23 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0517 06:45:09.815578 11664 catalog_manager.cc:3686] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 reported cstate change: config changed from index 5 to 6, NON_VOTER 63382e9ca0f7430ba027db6b03202ab2 (127.4.96.5) added. New cstate: current_term: 3 leader_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } W0517 06:45:09.820672 11726 consensus_peers.cc:434] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 63382e9ca0f7430ba027db6b03202ab2 (127.4.96.5:46864): Couldn't send request to peer 63382e9ca0f7430ba027db6b03202ab2 for tablet 1ab1a102dd9c49aba0780da8d8e19b23. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ab1a102dd9c49aba0780da8d8e19b23. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:45:09.851204 11643 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94 with a delay of 106 ms (attempt = 3) W0517 06:45:09.851390 11643 catalog_manager.cc:2810] Async tablet task 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a0e9f9fef615453a89e52dc146476b94 I0517 06:45:09.853518 11645 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24 with a delay of 96 ms (attempt = 3) W0517 06:45:09.853678 11645 catalog_manager.cc:2810] Async tablet task 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 432b3fcdbb4c4dc0aa368a7f02647c24 I0517 06:45:09.950232 11645 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24 with a delay of 174 ms (attempt = 4) W0517 06:45:09.950381 11645 catalog_manager.cc:2810] Async tablet task 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 432b3fcdbb4c4dc0aa368a7f02647c24 I0517 06:45:09.957947 11643 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94 with a delay of 157 ms (attempt = 4) W0517 06:45:09.958163 11643 catalog_manager.cc:2810] Async tablet task 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a0e9f9fef615453a89e52dc146476b94 I0517 06:45:10.115844 11643 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94 with a delay of 277 ms (attempt = 5) W0517 06:45:10.116003 11643 catalog_manager.cc:2810] Async tablet task 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a0e9f9fef615453a89e52dc146476b94 I0517 06:45:10.125165 11645 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24 with a delay of 276 ms (attempt = 5) W0517 06:45:10.125340 11645 catalog_manager.cc:2810] Async tablet task 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 432b3fcdbb4c4dc0aa368a7f02647c24 I0517 06:45:10.346164 11858 ts_tablet_manager.cc:669] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: Initiating tablet copy from peer 6c3dcb1ce66b4b3a8e14a7305a1af218 (127.4.96.1:49563) I0517 06:45:10.346447 11858 tablet_copy_client.cc:239] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.1:49563 I0517 06:45:10.355348 11815 tablet_copy_service.cc:133] P 6c3dcb1ce66b4b3a8e14a7305a1af218: Received BeginTabletCopySession request for tablet 1ab1a102dd9c49aba0780da8d8e19b23 from peer 63382e9ca0f7430ba027db6b03202ab2 ({username='slave'} at 127.4.96.5:60360) I0517 06:45:10.355634 11815 tablet_copy_service.cc:154] P 6c3dcb1ce66b4b3a8e14a7305a1af218: Beginning new tablet copy session on tablet 1ab1a102dd9c49aba0780da8d8e19b23 from peer 63382e9ca0f7430ba027db6b03202ab2 at {username='slave'} at 127.4.96.5:60360: session id = 63382e9ca0f7430ba027db6b03202ab2-1ab1a102dd9c49aba0780da8d8e19b23 I0517 06:45:10.358764 11815 tablet_copy_source_session.cc:202] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:45:10.361390 11858 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1ab1a102dd9c49aba0780da8d8e19b23. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:10.372089 11858 ts_tablet_manager.cc:1173] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:45:10.372455 11858 tablet_copy_client.cc:565] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: tablet copy: Starting download of 0 data blocks... I0517 06:45:10.372685 11858 tablet_copy_client.cc:528] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: tablet copy: Starting download of 1 WAL segments... I0517 06:45:10.374729 11858 tablet_copy_client.cc:404] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:45:10.377408 11858 ts_tablet_manager.cc:1028] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: Bootstrapping tablet I0517 06:45:10.377737 11858 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: Bootstrap starting. I0517 06:45:10.383152 11858 log.cc:520] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: Log is configured to *not* fsync() on all Append() calls I0517 06:45:10.389142 11858 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:45:10.389598 11858 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: Bootstrap complete. I0517 06:45:10.389904 11858 ts_tablet_manager.cc:1045] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: Time spent bootstrapping tablet: real 0.012s user 0.012s sys 0.000s I0517 06:45:10.393959 11643 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94 with a delay of 546 ms (attempt = 6) W0517 06:45:10.394160 11643 catalog_manager.cc:2810] Async tablet task 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a0e9f9fef615453a89e52dc146476b94 I0517 06:45:10.399153 11858 raft_consensus.cc:303] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: true } } unsafe_config_change: true I0517 06:45:10.399718 11858 raft_consensus.cc:604] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 63382e9ca0f7430ba027db6b03202ab2, State: Initialized, Role: LEARNER I0517 06:45:10.400027 11858 consensus_queue.cc:227] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: true } } unsafe_config_change: true I0517 06:45:10.402115 11858 ts_tablet_manager.cc:1073] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s I0517 06:45:10.402213 11645 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24 with a delay of 534 ms (attempt = 6) W0517 06:45:10.402459 11645 catalog_manager.cc:2810] Async tablet task 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 432b3fcdbb4c4dc0aa368a7f02647c24 I0517 06:45:10.403015 11815 tablet_copy_service.cc:323] P 6c3dcb1ce66b4b3a8e14a7305a1af218: Request end of tablet copy session 63382e9ca0f7430ba027db6b03202ab2-1ab1a102dd9c49aba0780da8d8e19b23 received from {username='slave'} at 127.4.96.5:60360 I0517 06:45:10.403201 11815 tablet_copy_service.cc:411] P 6c3dcb1ce66b4b3a8e14a7305a1af218: Ending tablet copy session 63382e9ca0f7430ba027db6b03202ab2-1ab1a102dd9c49aba0780da8d8e19b23 on tablet 1ab1a102dd9c49aba0780da8d8e19b23 with peer 63382e9ca0f7430ba027db6b03202ab2 I0517 06:45:10.423682 11863 ts_tablet_manager.cc:669] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: Initiating tablet copy from peer 6c3dcb1ce66b4b3a8e14a7305a1af218 (127.4.96.1:49563) I0517 06:45:10.423946 11863 tablet_copy_client.cc:239] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.1:49563 I0517 06:45:10.432572 11815 tablet_copy_service.cc:133] P 6c3dcb1ce66b4b3a8e14a7305a1af218: Received BeginTabletCopySession request for tablet 1ab1a102dd9c49aba0780da8d8e19b23 from peer 23a5a54ae9484026bfe44987841a7b20 ({username='slave'} at 127.4.96.4:58212) I0517 06:45:10.432788 11815 tablet_copy_service.cc:154] P 6c3dcb1ce66b4b3a8e14a7305a1af218: Beginning new tablet copy session on tablet 1ab1a102dd9c49aba0780da8d8e19b23 from peer 23a5a54ae9484026bfe44987841a7b20 at {username='slave'} at 127.4.96.4:58212: session id = 23a5a54ae9484026bfe44987841a7b20-1ab1a102dd9c49aba0780da8d8e19b23 I0517 06:45:10.435638 11815 tablet_copy_source_session.cc:202] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:45:10.437918 11863 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1ab1a102dd9c49aba0780da8d8e19b23. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:10.448415 11863 ts_tablet_manager.cc:1173] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:45:10.448777 11863 tablet_copy_client.cc:565] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: tablet copy: Starting download of 0 data blocks... I0517 06:45:10.449075 11863 tablet_copy_client.cc:528] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: tablet copy: Starting download of 1 WAL segments... I0517 06:45:10.451102 11863 tablet_copy_client.cc:404] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:45:10.453845 11863 ts_tablet_manager.cc:1028] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: Bootstrapping tablet I0517 06:45:10.454210 11863 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: Bootstrap starting. I0517 06:45:10.459326 11863 log.cc:520] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: Log is configured to *not* fsync() on all Append() calls I0517 06:45:10.470263 11863 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:45:10.470702 11863 tablet_bootstrap.cc:436] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: Bootstrap complete. I0517 06:45:10.470995 11863 ts_tablet_manager.cc:1045] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s I0517 06:45:10.480070 11863 raft_consensus.cc:303] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: true } } unsafe_config_change: true I0517 06:45:10.480612 11863 raft_consensus.cc:604] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 23a5a54ae9484026bfe44987841a7b20, State: Initialized, Role: LEARNER I0517 06:45:10.480900 11863 consensus_queue.cc:227] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: NON_VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: true } } unsafe_config_change: true I0517 06:45:10.482825 11863 ts_tablet_manager.cc:1073] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s I0517 06:45:10.483688 11815 tablet_copy_service.cc:323] P 6c3dcb1ce66b4b3a8e14a7305a1af218: Request end of tablet copy session 23a5a54ae9484026bfe44987841a7b20-1ab1a102dd9c49aba0780da8d8e19b23 received from {username='slave'} at 127.4.96.4:58212 I0517 06:45:10.483855 11815 tablet_copy_service.cc:411] P 6c3dcb1ce66b4b3a8e14a7305a1af218: Ending tablet copy session 23a5a54ae9484026bfe44987841a7b20-1ab1a102dd9c49aba0780da8d8e19b23 on tablet 1ab1a102dd9c49aba0780da8d8e19b23 with peer 23a5a54ae9484026bfe44987841a7b20 W0517 06:45:10.576784 11574 tablet.cc:1678] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:45:10.649447 11451 tablet.cc:1678] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:10.927641 11530 raft_consensus.cc:1002] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0517 06:45:10.936825 11645 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24 with a delay of 1051 ms (attempt = 7) W0517 06:45:10.936992 11645 catalog_manager.cc:2810] Async tablet task 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=432b3fcdbb4c4dc0aa368a7f02647c24 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 432b3fcdbb4c4dc0aa368a7f02647c24 I0517 06:45:10.940840 11643 catalog_manager.cc:2791] Scheduling retry of 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94 with a delay of 1058 ms (attempt = 7) W0517 06:45:10.941020 11643 catalog_manager.cc:2810] Async tablet task 1ab1a102dd9c49aba0780da8d8e19b23 Delete Tablet RPC for TS=a0e9f9fef615453a89e52dc146476b94 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a0e9f9fef615453a89e52dc146476b94 I0517 06:45:11.023922 11406 raft_consensus.cc:1002] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0517 06:45:11.323945 11869 raft_consensus.cc:866] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: attempting to promote NON_VOTER 63382e9ca0f7430ba027db6b03202ab2 to VOTER I0517 06:45:11.324614 11869 consensus_peers.cc:456] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 63382e9ca0f7430ba027db6b03202ab2 (127.4.96.5:46864): Closing peer: 63382e9ca0f7430ba027db6b03202ab2 I0517 06:45:11.324765 11869 consensus_peers.cc:456] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 23a5a54ae9484026bfe44987841a7b20 (127.4.96.4:43655): Closing peer: 23a5a54ae9484026bfe44987841a7b20 I0517 06:45:11.325178 11869 consensus_queue.cc:204] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: false } } unsafe_config_change: true I0517 06:45:11.328574 11530 raft_consensus.cc:1057] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2 [term 3 LEARNER]: Refusing update from remote peer 6c3dcb1ce66b4b3a8e14a7305a1af218: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:45:11.329100 11406 raft_consensus.cc:1057] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20 [term 3 LEARNER]: Refusing update from remote peer 6c3dcb1ce66b4b3a8e14a7305a1af218: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0517 06:45:11.329694 11868 consensus_queue.cc:963] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:45:11.330253 11867 consensus_queue.cc:963] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0517 06:45:11.336051 11869 raft_consensus.cc:2658] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 63382e9ca0f7430ba027db6b03202ab2 (127.4.96.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: false } } unsafe_config_change: true } I0517 06:45:11.337141 11406 raft_consensus.cc:2658] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20 [term 3 LEARNER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 63382e9ca0f7430ba027db6b03202ab2 (127.4.96.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: false } } unsafe_config_change: true } I0517 06:45:11.337335 11530 raft_consensus.cc:2658] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 63382e9ca0f7430ba027db6b03202ab2 (127.4.96.5) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: false } } unsafe_config_change: true } I0517 06:45:11.343703 11867 raft_consensus.cc:866] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218: attempting to promote NON_VOTER 23a5a54ae9484026bfe44987841a7b20 to VOTER I0517 06:45:11.343209 11664 catalog_manager.cc:3686] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20 reported cstate change: config changed from index 6 to 7, 63382e9ca0f7430ba027db6b03202ab2 (127.4.96.5) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: true } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: false } } unsafe_config_change: true } I0517 06:45:11.344277 11867 consensus_peers.cc:456] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 63382e9ca0f7430ba027db6b03202ab2 (127.4.96.5:46864): Closing peer: 63382e9ca0f7430ba027db6b03202ab2 I0517 06:45:11.344405 11867 consensus_peers.cc:456] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 -> Peer 23a5a54ae9484026bfe44987841a7b20 (127.4.96.4:43655): Closing peer: 23a5a54ae9484026bfe44987841a7b20 I0517 06:45:11.344699 11867 consensus_queue.cc:204] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: false } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: false } } unsafe_config_change: true I0517 06:45:11.347404 11868 raft_consensus.cc:829] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 3 LEADER]: attempt to promote peer 23a5a54ae9484026bfe44987841a7b20: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing. I0517 06:45:11.347956 11530 raft_consensus.cc:1057] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2 [term 3 FOLLOWER]: Refusing update from remote peer 6c3dcb1ce66b4b3a8e14a7305a1af218: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0517 06:45:11.348929 11867 consensus_queue.cc:963] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0517 06:45:11.349552 11406 raft_consensus.cc:1057] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20 [term 3 LEARNER]: Refusing update from remote peer 6c3dcb1ce66b4b3a8e14a7305a1af218: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0517 06:45:11.350497 11867 consensus_queue.cc:963] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0517 06:45:11.353312 11869 raft_consensus.cc:2658] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 23a5a54ae9484026bfe44987841a7b20 (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: false } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: false } } unsafe_config_change: true } I0517 06:45:11.354467 11530 raft_consensus.cc:2658] T 1ab1a102dd9c49aba0780da8d8e19b23 P 63382e9ca0f7430ba027db6b03202ab2 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 23a5a54ae9484026bfe44987841a7b20 (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: false } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: false } } unsafe_config_change: true } I0517 06:45:11.355742 11405 raft_consensus.cc:2658] T 1ab1a102dd9c49aba0780da8d8e19b23 P 23a5a54ae9484026bfe44987841a7b20 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 23a5a54ae9484026bfe44987841a7b20 (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: false } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: false } } unsafe_config_change: true } I0517 06:45:11.360232 11662 catalog_manager.cc:3686] T 1ab1a102dd9c49aba0780da8d8e19b23 P 6c3dcb1ce66b4b3a8e14a7305a1af218 reported cstate change: config changed from index 7 to 8, 23a5a54ae9484026bfe44987841a7b20 (127.4.96.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 49563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 43655 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 46864 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0517 06:45:11.463752 4480 kudu-admin-test.cc:1017] Tablet locations: tablet_id: "1ab1a102dd9c49aba0780da8d8e19b23" replicas { ts_info { permanent_uuid: "6c3dcb1ce66b4b3a8e14a7305a1af218" rpc_addresses { host: "127.4.96.1" port: 49563 } } role: LEADER } replicas { ts_info { permanent_uuid: "23a5a54ae9484026bfe44987841a7b20" rpc_addresses { host: "127.4.96.4" port: 43655 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "63382e9ca0f7430ba027db6b03202ab2" rpc_addresses { host: "127.4.96.5" port: 46864 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0517 06:45:11.465302 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 11714 W0517 06:45:11.487392 11648 connection.cc:512] server connection from 127.4.96.1:54248 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:54248): Connection reset by peer (error 104) W0517 06:45:11.487401 10879 connection.cc:512] client connection to 127.4.96.1:49563 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:49563): Connection reset by peer (error 104) W0517 06:45:11.488101 11644 connection.cc:512] client connection to 127.4.96.1:49563 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:49563): Connection reset by peer (error 104) W0517 06:45:11.488204 11461 connection.cc:512] client connection to 127.4.96.1:49563 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:49563): Connection reset by peer (error 104) W0517 06:45:11.489135 11336 connection.cc:512] server connection from 127.4.96.1:53370 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:53370): Connection reset by peer (error 104) W0517 06:45:11.489143 11460 connection.cc:512] server connection from 127.4.96.1:48843 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:48843): Connection reset by peer (error 104) W0517 06:45:11.490011 11337 connection.cc:512] client connection to 127.4.96.1:49563 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:49563): Connection reset by peer (error 104) I0517 06:45:11.491014 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 11328 W0517 06:45:11.508824 11645 connection.cc:512] server connection from 127.4.96.4:57213 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:57213): Connection reset by peer (error 104) W0517 06:45:11.508879 10882 connection.cc:512] client connection to 127.4.96.4:43655 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:43655): Connection reset by peer (error 104) W0517 06:45:11.509737 10880 connection.cc:512] client connection to 127.4.96.4:43655 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:43655): Connection reset by peer (error 104) I0517 06:45:11.510576 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 11450 W0517 06:45:11.535163 11644 connection.cc:512] server connection from 127.4.96.5:55411 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:55411): Connection reset by peer (error 104) W0517 06:45:11.535156 10882 connection.cc:512] client connection to 127.4.96.5:46864 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:46864): Connection reset by peer (error 104) I0517 06:45:11.535987 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 11635 W0517 06:45:11.560912 10882 connection.cc:512] client connection to 127.4.96.62:60575 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:60575): Connection reset by peer (error 104) [ OK ] AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL (12559 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs I0517 06:45:11.576222 4480 test_util.cc:200] Using random seed: 760395401 I0517 06:45:11.579174 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:45:11.579277 4480 ts_itest-base.cc:114] -------------- I0517 06:45:11.579373 4480 ts_itest-base.cc:115] 9 tablet servers I0517 06:45:11.579463 4480 ts_itest-base.cc:116] 5 replicas per TS I0517 06:45:11.579553 4480 ts_itest-base.cc:117] -------------- I0517 06:45:11.581709 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:50584 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:11.694643 11888 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:11.694949 11888 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:11.703779 11888 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:45:11.703896 11888 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:11.703995 11888 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:45:11.704105 11888 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:45:11.717933 11888 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:50584 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:11.725551 11888 master_main.cc:80] Initializing master server... I0517 06:45:11.727067 11888 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 154819us I0517 06:45:11.727807 11888 fs_manager.cc:260] Metadata directory not provided I0517 06:45:11.727927 11888 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:11.728119 11888 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:45:11.728221 11888 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:11.735311 11888 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "8262fed3fddc45008f1ee9e8f98327fa" format_stamp: "Formatted at 2018-05-17 06:45:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:11.735962 11888 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "8262fed3fddc45008f1ee9e8f98327fa" format_stamp: "Formatted at 2018-05-17 06:45:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:11.745460 11888 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.012s sys 0.000s I0517 06:45:11.746598 11888 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:11.746978 11888 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:11.753279 11888 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:45:11.753386 11888 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "8262fed3fddc45008f1ee9e8f98327fa" format_stamp: "Formatted at 2018-05-17 06:45:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:11.753573 11888 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:11.924652 11888 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:11.926306 11888 master_main.cc:83] Starting Master server... I0517 06:45:11.965865 11888 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:50584 I0517 06:45:11.966747 11888 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:45:11.966876 11888 webserver.cc:184] Document root disabled I0517 06:45:11.968088 11888 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:58519/ I0517 06:45:11.969128 11888 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:45:11.971634 11949 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:11.974840 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 11888 I0517 06:45:11.976219 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:11.987732 11949 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa: Bootstrap starting. I0517 06:45:11.989939 11949 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa: No blocks or log segments found. Creating new log. I0517 06:45:11.991078 11949 log.cc:520] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa: Log is configured to *not* fsync() on all Append() calls I0517 06:45:11.993350 11949 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa: No bootstrap required, opened a new log I0517 06:45:12.004927 11949 raft_consensus.cc:303] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8262fed3fddc45008f1ee9e8f98327fa" member_type: VOTER } I0517 06:45:12.005228 11949 raft_consensus.cc:329] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:12.005352 11949 raft_consensus.cc:604] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8262fed3fddc45008f1ee9e8f98327fa, State: Initialized, Role: FOLLOWER I0517 06:45:12.005786 11949 consensus_queue.cc:227] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8262fed3fddc45008f1ee9e8f98327fa" member_type: VOTER } I0517 06:45:12.006237 11949 raft_consensus.cc:343] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:45:12.006423 11949 raft_consensus.cc:435] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:45:12.006575 11949 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa: Snoozing failure detection for 1.501s (starting election) I0517 06:45:12.006718 11949 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:12.009104 11949 raft_consensus.cc:457] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8262fed3fddc45008f1ee9e8f98327fa" member_type: VOTER } I0517 06:45:12.009500 11949 leader_election.cc:270] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:12.011067 11952 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa: Snoozing failure detection for 2.092s (election complete) I0517 06:45:12.011328 11952 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:12.018447 11952 raft_consensus.cc:571] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [term 1 LEADER]: Becoming Leader. State: Replica: 8262fed3fddc45008f1ee9e8f98327fa, State: Running, Role: LEADER I0517 06:45:12.018847 11952 consensus_queue.cc:204] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8262fed3fddc45008f1ee9e8f98327fa" member_type: VOTER } I0517 06:45:12.021899 11949 sys_catalog.cc:335] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "8262fed3fddc45008f1ee9e8f98327fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8262fed3fddc45008f1ee9e8f98327fa" member_type: VOTER } } I0517 06:45:12.022320 11949 sys_catalog.cc:338] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [sys.catalog]: This master's current role is: LEADER WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:12.120728 11950 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:12.121184 11950 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:12.135149 11950 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:12.135614 11950 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:45:12.167824 11950 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:12.180953 11950 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:12.183336 11950 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 155319us I0517 06:45:12.183923 11950 fs_manager.cc:260] Metadata directory not provided I0517 06:45:12.184075 11950 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:12.184321 11950 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:45:12.184478 11950 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:12.028322 11954 sys_catalog.cc:335] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8262fed3fddc45008f1ee9e8f98327fa. Latest consensus state: current_term: 1 leader_uuid: "8262fed3fddc45008f1ee9e8f98327fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8262fed3fddc45008f1ee9e8f98327fa" member_type: VOTER } } I0517 06:45:12.189224 11954 sys_catalog.cc:338] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [sys.catalog]: This master's current role is: LEADER I0517 06:45:12.191145 11949 sys_catalog.cc:422] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [sys.catalog]: configured and running, proceeding with master startup. I0517 06:45:12.039847 11953 sys_catalog.cc:335] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8262fed3fddc45008f1ee9e8f98327fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8262fed3fddc45008f1ee9e8f98327fa" member_type: VOTER } } I0517 06:45:12.194360 11953 sys_catalog.cc:338] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa [sys.catalog]: This master's current role is: LEADER I0517 06:45:12.196295 11888 master_main.cc:86] Master server successfully started. I0517 06:45:12.196552 11950 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "3c4ca03e934448e789a3413ea92ab6c6" format_stamp: "Formatted at 2018-05-17 06:45:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:12.197239 11950 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "3c4ca03e934448e789a3413ea92ab6c6" format_stamp: "Formatted at 2018-05-17 06:45:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:12.202190 11957 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:45:12.206626 11957 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:45:12.207491 11950 fs_manager.cc:495] Time spent creating directory manager: real 0.010s user 0.004s sys 0.004s I0517 06:45:12.209164 11950 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:12.209717 11950 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:12.220224 11950 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:45:12.220386 11950 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "3c4ca03e934448e789a3413ea92ab6c6" format_stamp: "Formatted at 2018-05-17 06:45:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:12.220674 11950 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:12.294327 11957 catalog_manager.cc:899] Generated new certificate authority record I0517 06:45:12.296592 11957 catalog_manager.cc:1021] Loading token signing keys... I0517 06:45:12.317780 11957 catalog_manager.cc:3962] T 00000000000000000000000000000000 P 8262fed3fddc45008f1ee9e8f98327fa: Generated new TSK 0 I0517 06:45:12.458910 11950 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:12.462123 11950 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:12.463235 11950 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:12.594105 11950 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:38160 I0517 06:45:12.595002 11950 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:45:12.595114 11950 webserver.cc:184] Document root disabled I0517 06:45:12.596493 11950 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:58779/ I0517 06:45:12.597205 11950 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:45:12.602216 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 11950 I0517 06:45:12.603422 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:12.611820 11950 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:12.629477 12082 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50584 I0517 06:45:12.629776 12082 heartbeater.cc:426] Registering TS with master... I0517 06:45:12.630352 12082 heartbeater.cc:475] Master 127.4.96.62:50584 requested a full tablet report, sending... I0517 06:45:12.632395 11916 ts_manager.cc:81] Registered new tserver with Master: 3c4ca03e934448e789a3413ea92ab6c6 (127.4.96.1:38160) I0517 06:45:12.635079 11916 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:43461 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:12.715028 12086 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:12.715370 12086 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:12.723390 12086 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:12.723723 12086 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:45:12.736377 12086 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:12.744766 12086 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:12.746582 12086 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 155319us I0517 06:45:12.747014 12086 fs_manager.cc:260] Metadata directory not provided I0517 06:45:12.747138 12086 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:12.747344 12086 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:45:12.747462 12086 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:12.754999 12086 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "b63d0bdace5e4d598392de0641274b6c" format_stamp: "Formatted at 2018-05-17 06:45:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:12.755689 12086 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "b63d0bdace5e4d598392de0641274b6c" format_stamp: "Formatted at 2018-05-17 06:45:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:12.765149 12086 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:45:12.766332 12086 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:12.766772 12086 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:12.773296 12086 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:45:12.773408 12086 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "b63d0bdace5e4d598392de0641274b6c" format_stamp: "Formatted at 2018-05-17 06:45:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:12.773627 12086 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:13.024264 12086 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:13.027093 12086 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:13.028224 12086 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:13.131898 12086 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:51044 I0517 06:45:13.132881 12086 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:45:13.132982 12086 webserver.cc:184] Document root disabled I0517 06:45:13.134461 12086 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:54536/ I0517 06:45:13.135352 12086 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:45:13.138515 12086 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:13.141723 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 12086 I0517 06:45:13.142745 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:13.157511 12207 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50584 I0517 06:45:13.157755 12207 heartbeater.cc:426] Registering TS with master... I0517 06:45:13.158246 12207 heartbeater.cc:475] Master 127.4.96.62:50584 requested a full tablet report, sending... I0517 06:45:13.159474 11916 ts_manager.cc:81] Registered new tserver with Master: b63d0bdace5e4d598392de0641274b6c (127.4.96.2:51044) I0517 06:45:13.160539 11916 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:37185 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:13.260731 12211 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:13.261116 12211 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:13.269471 12211 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:13.269809 12211 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:45:13.283439 12211 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:13.292632 12211 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:13.294802 12211 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 155819us I0517 06:45:13.295307 12211 fs_manager.cc:260] Metadata directory not provided I0517 06:45:13.295434 12211 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:13.295653 12211 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:45:13.295783 12211 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:13.303910 12211 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "b2afd3be3c104d17802599476b02d509" format_stamp: "Formatted at 2018-05-17 06:45:13 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:13.304635 12211 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "b2afd3be3c104d17802599476b02d509" format_stamp: "Formatted at 2018-05-17 06:45:13 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:13.314869 12211 fs_manager.cc:495] Time spent creating directory manager: real 0.010s user 0.008s sys 0.004s I0517 06:45:13.316087 12211 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:13.316545 12211 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:13.323415 12211 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0517 06:45:13.323529 12211 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "b2afd3be3c104d17802599476b02d509" format_stamp: "Formatted at 2018-05-17 06:45:13 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:13.323752 12211 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:13.488831 12211 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:13.491547 12211 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:13.492650 12211 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:13.585851 12211 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:49957 I0517 06:45:13.586841 12211 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:45:13.586943 12211 webserver.cc:184] Document root disabled I0517 06:45:13.588410 12211 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:32977/ I0517 06:45:13.589298 12211 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:45:13.589776 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 12211 I0517 06:45:13.590867 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:13.592278 12211 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:13.616502 12332 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50584 I0517 06:45:13.616756 12332 heartbeater.cc:426] Registering TS with master... I0517 06:45:13.617213 12332 heartbeater.cc:475] Master 127.4.96.62:50584 requested a full tablet report, sending... I0517 06:45:13.618510 11916 ts_manager.cc:81] Registered new tserver with Master: b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957) I0517 06:45:13.619601 11916 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:53598 I0517 06:45:13.638269 12082 heartbeater.cc:467] Master 127.4.96.62:50584 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:13.697599 12334 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:13.697908 12334 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:13.705740 12334 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:13.706064 12334 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:45:13.718452 12334 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:13.726538 12334 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:13.728301 12334 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 155819us I0517 06:45:13.728744 12334 fs_manager.cc:260] Metadata directory not provided I0517 06:45:13.728857 12334 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:13.729046 12334 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:45:13.729168 12334 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:13.736637 12334 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "bc2cfadffd3547ae88f42510ed8395f3" format_stamp: "Formatted at 2018-05-17 06:45:13 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:13.737278 12334 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "bc2cfadffd3547ae88f42510ed8395f3" format_stamp: "Formatted at 2018-05-17 06:45:13 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:13.746779 12334 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:45:13.747876 12334 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:13.748265 12334 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:13.754804 12334 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0517 06:45:13.754914 12334 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "bc2cfadffd3547ae88f42510ed8395f3" format_stamp: "Formatted at 2018-05-17 06:45:13 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:13.755110 12334 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:13.908593 12334 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:13.910939 12334 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:13.911844 12334 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:14.000779 12334 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:39443 I0517 06:45:14.001653 12334 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:45:14.001749 12334 webserver.cc:184] Document root disabled I0517 06:45:14.002941 12334 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:39819/ I0517 06:45:14.003636 12334 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:45:14.006700 12334 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:14.007452 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 12334 I0517 06:45:14.008467 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.4.96.5:0 --local_ip_for_outbound_sockets=127.4.96.5 --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:14.025249 12456 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50584 I0517 06:45:14.025524 12456 heartbeater.cc:426] Registering TS with master... I0517 06:45:14.026084 12456 heartbeater.cc:475] Master 127.4.96.62:50584 requested a full tablet report, sending... I0517 06:45:14.027659 11916 ts_manager.cc:81] Registered new tserver with Master: bc2cfadffd3547ae88f42510ed8395f3 (127.4.96.4:39443) I0517 06:45:14.028743 11916 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:49030 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:14.115324 12459 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:14.115653 12459 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:14.123545 12459 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:14.123888 12459 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.5 I0517 06:45:14.136128 12459 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.5 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:14.144264 12459 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:14.146044 12459 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 156319us I0517 06:45:14.146474 12459 fs_manager.cc:260] Metadata directory not provided I0517 06:45:14.146603 12459 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:14.146801 12459 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0517 06:45:14.146920 12459 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:14.154326 12459 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "6f7859ab255d4820bf262d97c42cdd80" format_stamp: "Formatted at 2018-05-17 06:45:14 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:14.154979 12459 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "6f7859ab255d4820bf262d97c42cdd80" format_stamp: "Formatted at 2018-05-17 06:45:14 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:14.162731 12207 heartbeater.cc:467] Master 127.4.96.62:50584 was elected leader, sending a full tablet report... I0517 06:45:14.164144 12459 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:45:14.165427 12459 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:14.165841 12459 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:14.172426 12459 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.004s I0517 06:45:14.172539 12459 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal uuid: "6f7859ab255d4820bf262d97c42cdd80" format_stamp: "Formatted at 2018-05-17 06:45:14 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:14.172744 12459 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:14.352380 12459 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:14.354728 12459 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:14.355703 12459 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:14.440235 12459 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.5:59751 I0517 06:45:14.441117 12459 webserver.cc:173] Starting webserver on 127.4.96.5:0 I0517 06:45:14.441226 12459 webserver.cc:184] Document root disabled I0517 06:45:14.442400 12459 webserver.cc:311] Webserver started. Bound to: http://127.4.96.5:58412/ I0517 06:45:14.443095 12459 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb I0517 06:45:14.446395 12459 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:14.446816 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 12459 I0517 06:45:14.447741 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.4.96.6:0 --local_ip_for_outbound_sockets=127.4.96.6 --webserver_interface=127.4.96.6 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:14.463047 12580 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50584 I0517 06:45:14.463276 12580 heartbeater.cc:426] Registering TS with master... I0517 06:45:14.463701 12580 heartbeater.cc:475] Master 127.4.96.62:50584 requested a full tablet report, sending... I0517 06:45:14.464994 11916 ts_manager.cc:81] Registered new tserver with Master: 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751) I0517 06:45:14.466140 11916 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.5:39442 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:14.554625 12583 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:14.554927 12583 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:14.562919 12583 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:14.563233 12583 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.6 I0517 06:45:14.575969 12583 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.6:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.4.96.6 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.6 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:14.583987 12583 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:14.585788 12583 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 156319us I0517 06:45:14.586237 12583 fs_manager.cc:260] Metadata directory not provided I0517 06:45:14.586369 12583 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:14.586563 12583 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data/instance: No such file or directory (error 2) I0517 06:45:14.586675 12583 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:14.594182 12583 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data/instance: uuid: "c7fc5683934e4c9f885de04addaa2076" format_stamp: "Formatted at 2018-05-17 06:45:14 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:14.594818 12583 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/wal/instance: uuid: "c7fc5683934e4c9f885de04addaa2076" format_stamp: "Formatted at 2018-05-17 06:45:14 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:14.604161 12583 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:45:14.605278 12583 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:14.605674 12583 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:14.612045 12583 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:45:14.612151 12583 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/wal uuid: "c7fc5683934e4c9f885de04addaa2076" format_stamp: "Formatted at 2018-05-17 06:45:14 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:14.612331 12583 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:14.621831 12332 heartbeater.cc:467] Master 127.4.96.62:50584 was elected leader, sending a full tablet report... I0517 06:45:14.769196 12583 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:14.771611 12583 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:14.772511 12583 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:14.856412 12583 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.6:43583 I0517 06:45:14.857291 12583 webserver.cc:173] Starting webserver on 127.4.96.6:0 I0517 06:45:14.857400 12583 webserver.cc:184] Document root disabled I0517 06:45:14.858603 12583 webserver.cc:311] Webserver started. Bound to: http://127.4.96.6:37509/ I0517 06:45:14.859267 12583 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-5/data/info.pb I0517 06:45:14.861774 12583 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:14.866750 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 12583 I0517 06:45:14.868234 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.4.96.7:0 --local_ip_for_outbound_sockets=127.4.96.7 --webserver_interface=127.4.96.7 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:14.881202 12705 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50584 I0517 06:45:14.881443 12705 heartbeater.cc:426] Registering TS with master... I0517 06:45:14.881883 12705 heartbeater.cc:475] Master 127.4.96.62:50584 requested a full tablet report, sending... I0517 06:45:14.883199 11916 ts_manager.cc:81] Registered new tserver with Master: c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583) I0517 06:45:14.884279 11916 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.6:56588 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:14.971462 12708 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:14.971771 12708 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:14.979684 12708 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:14.979991 12708 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.7 I0517 06:45:14.992377 12708 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.7:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.4.96.7 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.7 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:15.000211 12708 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:15.001942 12708 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 156319us I0517 06:45:15.002357 12708 fs_manager.cc:260] Metadata directory not provided I0517 06:45:15.002465 12708 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:15.002646 12708 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data/instance: No such file or directory (error 2) I0517 06:45:15.002756 12708 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:15.010407 12708 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data/instance: uuid: "0d0febd027bb4eba809921f54e8e8a4c" format_stamp: "Formatted at 2018-05-17 06:45:15 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:15.011034 12708 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/wal/instance: uuid: "0d0febd027bb4eba809921f54e8e8a4c" format_stamp: "Formatted at 2018-05-17 06:45:15 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:15.020350 12708 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:45:15.021466 12708 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:15.021845 12708 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:15.028266 12708 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:45:15.028379 12708 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/wal uuid: "0d0febd027bb4eba809921f54e8e8a4c" format_stamp: "Formatted at 2018-05-17 06:45:15 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:15.028570 12708 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:15.030963 12456 heartbeater.cc:467] Master 127.4.96.62:50584 was elected leader, sending a full tablet report... I0517 06:45:15.185417 12708 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:15.187831 12708 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:15.188730 12708 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:15.276573 12708 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.7:51391 I0517 06:45:15.277496 12708 webserver.cc:173] Starting webserver on 127.4.96.7:0 I0517 06:45:15.277606 12708 webserver.cc:184] Document root disabled I0517 06:45:15.278790 12708 webserver.cc:311] Webserver started. Bound to: http://127.4.96.7:45833/ I0517 06:45:15.279462 12708 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-6/data/info.pb I0517 06:45:15.282495 12708 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:15.284210 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 12708 I0517 06:45:15.285167 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data --block_manager=log --rpc_bind_addresses=127.4.96.8:0 --local_ip_for_outbound_sockets=127.4.96.8 --webserver_interface=127.4.96.8 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:15.299911 12829 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50584 I0517 06:45:15.300142 12829 heartbeater.cc:426] Registering TS with master... I0517 06:45:15.300571 12829 heartbeater.cc:475] Master 127.4.96.62:50584 requested a full tablet report, sending... I0517 06:45:15.301937 11916 ts_manager.cc:81] Registered new tserver with Master: 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391) I0517 06:45:15.303514 11916 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.7:44773 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:15.390522 12832 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:15.390831 12832 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:15.398633 12832 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:15.398942 12832 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.8 I0517 06:45:15.411093 12832 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.8:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data/info.pb --webserver_interface=127.4.96.8 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.8 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:15.419098 12832 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:15.420917 12832 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 156819us I0517 06:45:15.421325 12832 fs_manager.cc:260] Metadata directory not provided I0517 06:45:15.421437 12832 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:15.421622 12832 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data/instance: No such file or directory (error 2) I0517 06:45:15.421736 12832 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:15.429431 12832 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data/instance: uuid: "71d6b0fe0b45450a809f4a25eb6bae51" format_stamp: "Formatted at 2018-05-17 06:45:15 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:15.430069 12832 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/wal/instance: uuid: "71d6b0fe0b45450a809f4a25eb6bae51" format_stamp: "Formatted at 2018-05-17 06:45:15 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:15.439442 12832 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:45:15.440539 12832 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:15.440979 12832 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:15.447268 12832 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:45:15.447372 12832 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/wal uuid: "71d6b0fe0b45450a809f4a25eb6bae51" format_stamp: "Formatted at 2018-05-17 06:45:15 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:15.447592 12832 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:15.468454 12580 heartbeater.cc:467] Master 127.4.96.62:50584 was elected leader, sending a full tablet report... I0517 06:45:15.613490 12832 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:15.615829 12832 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:15.616720 12832 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:15.703862 12832 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.8:50961 I0517 06:45:15.704715 12832 webserver.cc:173] Starting webserver on 127.4.96.8:0 I0517 06:45:15.704807 12832 webserver.cc:184] Document root disabled I0517 06:45:15.705981 12832 webserver.cc:311] Webserver started. Bound to: http://127.4.96.8:46003/ I0517 06:45:15.706634 12832 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-7/data/info.pb I0517 06:45:15.709094 12832 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:15.713112 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 12832 I0517 06:45:15.714021 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/data --block_manager=log --rpc_bind_addresses=127.4.96.9:0 --local_ip_for_outbound_sockets=127.4.96.9 --webserver_interface=127.4.96.9 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:15.726308 12953 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50584 I0517 06:45:15.726543 12953 heartbeater.cc:426] Registering TS with master... I0517 06:45:15.726963 12953 heartbeater.cc:475] Master 127.4.96.62:50584 requested a full tablet report, sending... I0517 06:45:15.728281 11916 ts_manager.cc:81] Registered new tserver with Master: 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961) I0517 06:45:15.729352 11916 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.8:44093 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:15.815342 12956 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:15.815649 12956 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:15.823484 12956 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:15.823792 12956 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.9 I0517 06:45:15.836374 12956 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.9:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/data/info.pb --webserver_interface=127.4.96.9 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50584 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.9 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:15.844601 12956 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:15.846333 12956 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 156819us I0517 06:45:15.846747 12956 fs_manager.cc:260] Metadata directory not provided I0517 06:45:15.846853 12956 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:15.847059 12956 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/data/instance: No such file or directory (error 2) I0517 06:45:15.847172 12956 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:15.854650 12956 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/data/instance: uuid: "f46a2a66ce864668aa605848aea6f97e" format_stamp: "Formatted at 2018-05-17 06:45:15 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:15.855316 12956 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/wal/instance: uuid: "f46a2a66ce864668aa605848aea6f97e" format_stamp: "Formatted at 2018-05-17 06:45:15 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:15.864462 12956 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:45:15.865557 12956 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:15.865949 12956 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:15.872351 12956 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:45:15.872457 12956 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/wal uuid: "f46a2a66ce864668aa605848aea6f97e" format_stamp: "Formatted at 2018-05-17 06:45:15 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:15.872668 12956 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:15.886636 12705 heartbeater.cc:467] Master 127.4.96.62:50584 was elected leader, sending a full tablet report... I0517 06:45:16.045248 12956 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:16.047524 12956 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:16.048416 12956 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:16.132043 12956 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.9:37079 I0517 06:45:16.132896 12956 webserver.cc:173] Starting webserver on 127.4.96.9:0 I0517 06:45:16.132997 12956 webserver.cc:184] Document root disabled I0517 06:45:16.134182 12956 webserver.cc:311] Webserver started. Bound to: http://127.4.96.9:47449/ I0517 06:45:16.134861 12956 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480/raft_consensus-itest-cluster/ts-8/data/info.pb I0517 06:45:16.137590 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 12956 I0517 06:45:16.143872 12956 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:16.166801 13077 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50584 I0517 06:45:16.167047 13077 heartbeater.cc:426] Registering TS with master... I0517 06:45:16.167570 13077 heartbeater.cc:475] Master 127.4.96.62:50584 requested a full tablet report, sending... I0517 06:45:16.169023 11915 ts_manager.cc:81] Registered new tserver with Master: f46a2a66ce864668aa605848aea6f97e (127.4.96.9:37079) I0517 06:45:16.170193 11915 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.9:45347 I0517 06:45:16.171905 4480 external_mini_cluster.cc:482] 9 TS(s) registered with all masters I0517 06:45:16.190129 11916 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45198: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 5 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:45:16.232138 12890 tablet_service.cc:759] Processing CreateTablet for tablet 2344f0911d62405895a6c44edca703bc (table=TestTable [id=9db52a502a324e048d9511407c546687]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:16.232993 12890 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2344f0911d62405895a6c44edca703bc. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:16.248190 12269 tablet_service.cc:759] Processing CreateTablet for tablet 2344f0911d62405895a6c44edca703bc (table=TestTable [id=9db52a502a324e048d9511407c546687]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:16.249055 12269 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2344f0911d62405895a6c44edca703bc. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:16.250000 12890 ts_tablet_manager.cc:1173] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:16.251647 12642 tablet_service.cc:759] Processing CreateTablet for tablet 2344f0911d62405895a6c44edca703bc (table=TestTable [id=9db52a502a324e048d9511407c546687]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:16.253052 12642 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2344f0911d62405895a6c44edca703bc. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:16.263433 12766 tablet_service.cc:759] Processing CreateTablet for tablet 2344f0911d62405895a6c44edca703bc (table=TestTable [id=9db52a502a324e048d9511407c546687]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:16.264216 12766 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2344f0911d62405895a6c44edca703bc. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:16.265516 12269 ts_tablet_manager.cc:1173] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:16.267132 13101 ts_tablet_manager.cc:1028] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: Bootstrapping tablet I0517 06:45:16.267684 13101 tablet_bootstrap.cc:436] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: Bootstrap starting. I0517 06:45:16.270426 12642 ts_tablet_manager.cc:1173] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:16.270663 13101 tablet_bootstrap.cc:587] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: No blocks or log segments found. Creating new log. I0517 06:45:16.272119 13101 log.cc:520] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: Log is configured to *not* fsync() on all Append() calls I0517 06:45:16.275341 13102 ts_tablet_manager.cc:1028] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Bootstrapping tablet I0517 06:45:16.275591 13103 ts_tablet_manager.cc:1028] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: Bootstrapping tablet I0517 06:45:16.276130 13103 tablet_bootstrap.cc:436] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: Bootstrap starting. I0517 06:45:16.276223 12517 tablet_service.cc:759] Processing CreateTablet for tablet 2344f0911d62405895a6c44edca703bc (table=TestTable [id=9db52a502a324e048d9511407c546687]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:16.276558 13102 tablet_bootstrap.cc:436] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Bootstrap starting. I0517 06:45:16.277052 12517 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2344f0911d62405895a6c44edca703bc. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:16.279027 13102 tablet_bootstrap.cc:587] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: No blocks or log segments found. Creating new log. I0517 06:45:16.279175 13103 tablet_bootstrap.cc:587] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: No blocks or log segments found. Creating new log. I0517 06:45:16.280217 13102 log.cc:520] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Log is configured to *not* fsync() on all Append() calls I0517 06:45:16.280716 13103 log.cc:520] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: Log is configured to *not* fsync() on all Append() calls I0517 06:45:16.282163 13101 tablet_bootstrap.cc:436] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: No bootstrap required, opened a new log I0517 06:45:16.282467 13101 ts_tablet_manager.cc:1045] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s I0517 06:45:16.291195 13102 tablet_bootstrap.cc:436] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: No bootstrap required, opened a new log I0517 06:45:16.291586 13102 ts_tablet_manager.cc:1045] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s I0517 06:45:16.293781 12517 ts_tablet_manager.cc:1173] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:16.294176 13101 raft_consensus.cc:303] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.296130 13101 raft_consensus.cc:329] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:16.296353 13101 raft_consensus.cc:604] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d6b0fe0b45450a809f4a25eb6bae51, State: Initialized, Role: FOLLOWER I0517 06:45:16.296789 13103 tablet_bootstrap.cc:436] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: No bootstrap required, opened a new log I0517 06:45:16.297144 13103 ts_tablet_manager.cc:1045] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.008s I0517 06:45:16.297535 12766 ts_tablet_manager.cc:1173] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:16.297081 13101 consensus_queue.cc:227] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.302026 13107 ts_tablet_manager.cc:1028] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Bootstrapping tablet I0517 06:45:16.302740 13107 tablet_bootstrap.cc:436] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Bootstrap starting. I0517 06:45:16.306790 12953 heartbeater.cc:467] Master 127.4.96.62:50584 was elected leader, sending a full tablet report... I0517 06:45:16.308463 13102 raft_consensus.cc:303] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.309377 13102 raft_consensus.cc:329] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:16.309600 13102 raft_consensus.cc:604] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2afd3be3c104d17802599476b02d509, State: Initialized, Role: FOLLOWER I0517 06:45:16.310290 13102 consensus_queue.cc:227] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.314260 13103 raft_consensus.cc:303] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.316181 13103 raft_consensus.cc:329] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:16.316359 13103 raft_consensus.cc:604] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fc5683934e4c9f885de04addaa2076, State: Initialized, Role: FOLLOWER I0517 06:45:16.316835 13103 consensus_queue.cc:227] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.323230 13101 ts_tablet_manager.cc:1073] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: Time spent starting tablet: real 0.041s user 0.012s sys 0.008s I0517 06:45:16.326442 13109 ts_tablet_manager.cc:1028] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Bootstrapping tablet I0517 06:45:16.326845 13107 tablet_bootstrap.cc:587] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: No blocks or log segments found. Creating new log. I0517 06:45:16.326892 13109 tablet_bootstrap.cc:436] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Bootstrap starting. I0517 06:45:16.333600 13109 tablet_bootstrap.cc:587] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: No blocks or log segments found. Creating new log. I0517 06:45:16.334587 13107 log.cc:520] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Log is configured to *not* fsync() on all Append() calls I0517 06:45:16.335032 13109 log.cc:520] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Log is configured to *not* fsync() on all Append() calls I0517 06:45:16.338116 13103 ts_tablet_manager.cc:1073] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: Time spent starting tablet: real 0.041s user 0.012s sys 0.028s I0517 06:45:16.343082 13102 ts_tablet_manager.cc:1073] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Time spent starting tablet: real 0.051s user 0.020s sys 0.024s I0517 06:45:16.347690 13109 tablet_bootstrap.cc:436] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: No bootstrap required, opened a new log I0517 06:45:16.348044 13109 ts_tablet_manager.cc:1045] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.000s I0517 06:45:16.382712 13109 raft_consensus.cc:303] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.383587 13109 raft_consensus.cc:329] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:16.383810 13109 raft_consensus.cc:604] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7859ab255d4820bf262d97c42cdd80, State: Initialized, Role: FOLLOWER I0517 06:45:16.384516 13109 consensus_queue.cc:227] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.398579 13109 ts_tablet_manager.cc:1073] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Time spent starting tablet: real 0.050s user 0.024s sys 0.000s I0517 06:45:16.403712 13111 raft_consensus.cc:435] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:16.404000 13111 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: Snoozing failure detection for 1.719s (starting election) I0517 06:45:16.404287 13111 raft_consensus.cc:457] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.405802 13111 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961) I0517 06:45:16.406461 13111 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391) I0517 06:45:16.408772 13111 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751) I0517 06:45:16.409173 13111 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957) I0517 06:45:16.305709 12829 heartbeater.cc:467] Master 127.4.96.62:50584 was elected leader, sending a full tablet report... I0517 06:45:16.430723 13107 tablet_bootstrap.cc:436] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: No bootstrap required, opened a new log I0517 06:45:16.431125 13107 ts_tablet_manager.cc:1045] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Time spent bootstrapping tablet: real 0.129s user 0.000s sys 0.004s I0517 06:45:16.462240 13107 raft_consensus.cc:303] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.463147 13107 raft_consensus.cc:329] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:16.463325 13107 raft_consensus.cc:604] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d0febd027bb4eba809921f54e8e8a4c, State: Initialized, Role: FOLLOWER I0517 06:45:16.463860 13107 consensus_queue.cc:227] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.489814 13107 ts_tablet_manager.cc:1073] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Time spent starting tablet: real 0.059s user 0.024s sys 0.008s I0517 06:45:16.532290 13114 raft_consensus.cc:435] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) W0517 06:45:16.532444 12830 tablet.cc:1678] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:16.532549 13114 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Snoozing failure detection for 1.926s (starting election) I0517 06:45:16.532855 13114 raft_consensus.cc:457] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.534349 13114 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961) I0517 06:45:16.537247 13114 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583) I0517 06:45:16.538394 13114 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391) I0517 06:45:16.545578 13114 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957) I0517 06:45:16.552551 13110 raft_consensus.cc:435] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:16.552794 13110 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Snoozing failure detection for 2.126s (starting election) I0517 06:45:16.553047 13110 raft_consensus.cc:457] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.554435 13110 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961) I0517 06:45:16.554903 13110 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583) I0517 06:45:16.555198 13110 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391) I0517 06:45:16.557657 13110 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751) W0517 06:45:16.343439 12333 tablet.cc:1678] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:45:16.364687 12706 tablet.cc:1678] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:16.783068 13118 raft_consensus.cc:435] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:16.783416 13118 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Snoozing failure detection for 1.720s (starting election) I0517 06:45:16.783715 13118 raft_consensus.cc:457] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.785152 13118 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961) I0517 06:45:16.785655 13118 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583) I0517 06:45:16.833499 13118 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751) I0517 06:45:16.838361 12786 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "b2afd3be3c104d17802599476b02d509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0febd027bb4eba809921f54e8e8a4c" is_pre_election: true I0517 06:45:16.838968 12786 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Snoozing failure detection for 2.053s (vote granted) I0517 06:45:16.839180 12786 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2afd3be3c104d17802599476b02d509 in term 0. I0517 06:45:16.840744 12223 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391) I0517 06:45:16.839979 13118 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957) I0517 06:45:16.892900 12289 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "0d0febd027bb4eba809921f54e8e8a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2afd3be3c104d17802599476b02d509" is_pre_election: true I0517 06:45:16.893427 12289 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Snoozing failure detection for 1.538s (vote granted) I0517 06:45:16.893658 12289 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d0febd027bb4eba809921f54e8e8a4c in term 0. I0517 06:45:16.894810 12716 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 pre-election: Vote granted by peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957) W0517 06:45:16.447854 12581 tablet.cc:1678] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:16.905318 12537 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "c7fc5683934e4c9f885de04addaa2076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7859ab255d4820bf262d97c42cdd80" is_pre_election: true I0517 06:45:16.905850 12537 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Snoozing failure detection for 1.662s (vote granted) I0517 06:45:16.906163 12537 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7fc5683934e4c9f885de04addaa2076 in term 0. I0517 06:45:16.907419 12592 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751) I0517 06:45:16.918273 12289 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "c7fc5683934e4c9f885de04addaa2076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2afd3be3c104d17802599476b02d509" is_pre_election: true I0517 06:45:16.918694 12289 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Snoozing failure detection for 1.668s (vote granted) I0517 06:45:16.918871 12289 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7fc5683934e4c9f885de04addaa2076 in term 0. I0517 06:45:16.920210 12592 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 pre-election: Vote granted by peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957) I0517 06:45:16.920469 12592 leader_election.cc:270] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:45:16.925127 12786 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "c7fc5683934e4c9f885de04addaa2076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0febd027bb4eba809921f54e8e8a4c" is_pre_election: true I0517 06:45:16.925582 12786 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Snoozing failure detection for 1.753s (vote granted) I0517 06:45:16.925773 12786 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7fc5683934e4c9f885de04addaa2076 in term 0. I0517 06:45:16.926873 12596 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391) I0517 06:45:16.928514 12537 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "0d0febd027bb4eba809921f54e8e8a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7859ab255d4820bf262d97c42cdd80" is_pre_election: true I0517 06:45:16.928906 12537 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Snoozing failure detection for 1.694s (vote granted) I0517 06:45:16.929093 12537 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d0febd027bb4eba809921f54e8e8a4c in term 0. I0517 06:45:16.930167 12716 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751) I0517 06:45:16.930385 12716 leader_election.cc:270] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:45:16.930927 13118 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Snoozing failure detection for 2.225s (election complete) I0517 06:45:16.931133 13118 raft_consensus.cc:2511] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:45:16.931306 13118 raft_consensus.cc:435] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:45:16.931463 13118 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Snoozing failure detection for 1.901s (starting election) I0517 06:45:16.931622 13118 raft_consensus.cc:2756] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:16.936918 13118 raft_consensus.cc:457] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.938263 13118 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 election: Requesting vote from peer 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961) I0517 06:45:16.938905 13118 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 election: Requesting vote from peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583) I0517 06:45:16.939435 13118 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 election: Requesting vote from peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751) I0517 06:45:16.940153 13118 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 election: Requesting vote from peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957) I0517 06:45:16.941433 12289 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "0d0febd027bb4eba809921f54e8e8a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2afd3be3c104d17802599476b02d509" I0517 06:45:16.941521 12537 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "0d0febd027bb4eba809921f54e8e8a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7859ab255d4820bf262d97c42cdd80" I0517 06:45:16.941845 12289 raft_consensus.cc:2756] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:16.941906 12537 raft_consensus.cc:2756] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:16.942106 12289 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Snoozing failure detection for 1.651s (vote granted) I0517 06:45:16.942169 12537 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Snoozing failure detection for 1.955s (vote granted) I0517 06:45:16.947399 12289 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d0febd027bb4eba809921f54e8e8a4c in term 1. I0517 06:45:16.947293 12537 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d0febd027bb4eba809921f54e8e8a4c in term 1. I0517 06:45:16.948252 12716 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 election: Vote granted by peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957) I0517 06:45:16.948684 12716 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 election: Vote granted by peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751) I0517 06:45:16.948870 12716 leader_election.cc:270] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:16.950567 13118 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Snoozing failure detection for 2.060s (election complete) I0517 06:45:16.950793 13118 raft_consensus.cc:2511] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:16.951954 13118 raft_consensus.cc:571] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 LEADER]: Becoming Leader. State: Replica: 0d0febd027bb4eba809921f54e8e8a4c, State: Running, Role: LEADER I0517 06:45:16.952461 13118 consensus_queue.cc:204] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:16.967182 13141 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: Snoozing failure detection for 1.584s (election complete) W0517 06:45:16.967770 12592 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.96.3:49957, user_credentials={real_user=slave}} blocked reactor thread for 47570.9us I0517 06:45:16.969357 13141 raft_consensus.cc:2511] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:45:16.969867 13141 raft_consensus.cc:435] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:45:16.974768 13141 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: Snoozing failure detection for 1.945s (starting election) I0517 06:45:16.975188 13141 raft_consensus.cc:2756] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:16.976373 12662 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "b2afd3be3c104d17802599476b02d509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fc5683934e4c9f885de04addaa2076" is_pre_election: true W0517 06:45:16.461403 12954 tablet.cc:1678] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:16.981081 11916 catalog_manager.cc:3686] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c reported cstate change: term changed from 0 to 1, leader changed from to 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7). New cstate: current_term: 1 leader_uuid: "0d0febd027bb4eba809921f54e8e8a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } health_report { overall_health: UNKNOWN } } } I0517 06:45:17.004948 12289 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "6f7859ab255d4820bf262d97c42cdd80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2afd3be3c104d17802599476b02d509" is_pre_election: true I0517 06:45:17.005532 12289 raft_consensus.cc:2149] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f7859ab255d4820bf262d97c42cdd80 in current term 1: Already voted for candidate 0d0febd027bb4eba809921f54e8e8a4c in this term. I0517 06:45:17.006927 12467 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [CANDIDATE]: Term 1 pre-election: Vote denied by peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957). Message: Invalid argument: T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f7859ab255d4820bf262d97c42cdd80 in current term 1: Already voted for candidate 0d0febd027bb4eba809921f54e8e8a4c in this term. I0517 06:45:17.016690 12537 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "b2afd3be3c104d17802599476b02d509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7859ab255d4820bf262d97c42cdd80" is_pre_election: true I0517 06:45:17.017274 12537 raft_consensus.cc:2149] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2afd3be3c104d17802599476b02d509 in current term 1: Already voted for candidate 0d0febd027bb4eba809921f54e8e8a4c in this term. I0517 06:45:17.019433 12219 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751). Message: Invalid argument: T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2afd3be3c104d17802599476b02d509 in current term 1: Already voted for candidate 0d0febd027bb4eba809921f54e8e8a4c in this term. I0517 06:45:17.018857 12786 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "6f7859ab255d4820bf262d97c42cdd80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0febd027bb4eba809921f54e8e8a4c" is_pre_election: true I0517 06:45:17.019479 12910 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "0d0febd027bb4eba809921f54e8e8a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" is_pre_election: true I0517 06:45:17.019964 12786 raft_consensus.cc:2180] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f7859ab255d4820bf262d97c42cdd80 for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:45:17.020066 12910 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: Snoozing failure detection for 1.670s (vote granted) I0517 06:45:17.020292 12910 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d0febd027bb4eba809921f54e8e8a4c in term 0. I0517 06:45:17.021001 12471 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391). Message: Invalid argument: T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 6f7859ab255d4820bf262d97c42cdd80 for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:45:17.020967 12661 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "6f7859ab255d4820bf262d97c42cdd80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fc5683934e4c9f885de04addaa2076" is_pre_election: true I0517 06:45:17.021162 12909 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "0d0febd027bb4eba809921f54e8e8a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" I0517 06:45:17.021584 12909 raft_consensus.cc:2756] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:17.021806 12909 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: Snoozing failure detection for 2.147s (vote granted) I0517 06:45:17.006462 13141 raft_consensus.cc:457] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:17.025024 12662 raft_consensus.cc:2149] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2afd3be3c104d17802599476b02d509 in current term 1: Already voted for candidate c7fc5683934e4c9f885de04addaa2076 in this term. I0517 06:45:17.026206 12661 raft_consensus.cc:2194] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f7859ab255d4820bf262d97c42cdd80 for term 1 because replica is already servicing an update from a current leader or another vote. I0517 06:45:17.027928 12717 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961) I0517 06:45:17.028221 12467 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [CANDIDATE]: Term 1 pre-election: Vote denied by peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583). Message: Service unavailable: T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f7859ab255d4820bf262d97c42cdd80 for term 1 because replica is already servicing an update from a current leader or another vote. I0517 06:45:17.028455 12467 leader_election.cc:270] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0517 06:45:17.028980 13114 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Snoozing failure detection for 2.085s (election complete) I0517 06:45:17.029167 13114 raft_consensus.cc:2459] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0517 06:45:17.031217 12662 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "0d0febd027bb4eba809921f54e8e8a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fc5683934e4c9f885de04addaa2076" I0517 06:45:17.031714 12662 raft_consensus.cc:2149] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d0febd027bb4eba809921f54e8e8a4c in current term 1: Already voted for candidate c7fc5683934e4c9f885de04addaa2076 in this term. I0517 06:45:17.031908 12219 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [CANDIDATE]: Term 1 pre-election: Vote denied by peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583). Message: Invalid argument: T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2afd3be3c104d17802599476b02d509 in current term 1: Already voted for candidate c7fc5683934e4c9f885de04addaa2076 in this term. I0517 06:45:17.032655 12716 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 election: Vote denied by peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583). Message: Invalid argument: T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d0febd027bb4eba809921f54e8e8a4c in current term 1: Already voted for candidate c7fc5683934e4c9f885de04addaa2076 in this term. I0517 06:45:17.033174 12661 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "0d0febd027bb4eba809921f54e8e8a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fc5683934e4c9f885de04addaa2076" is_pre_election: true I0517 06:45:17.033649 12661 raft_consensus.cc:2149] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d0febd027bb4eba809921f54e8e8a4c in current term 1: Already voted for candidate c7fc5683934e4c9f885de04addaa2076 in this term. I0517 06:45:17.035097 12716 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 pre-election: Vote denied by peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583). Message: Invalid argument: T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d0febd027bb4eba809921f54e8e8a4c in current term 1: Already voted for candidate c7fc5683934e4c9f885de04addaa2076 in this term. I0517 06:45:17.035807 13141 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 election: Requesting vote from peer 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961) I0517 06:45:17.036592 13141 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 election: Requesting vote from peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391) I0517 06:45:17.036711 12909 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d0febd027bb4eba809921f54e8e8a4c in term 1. I0517 06:45:17.037447 13141 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 election: Requesting vote from peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751) I0517 06:45:17.037608 12717 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [CANDIDATE]: Term 1 election: Vote granted by peer 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961) I0517 06:45:17.038535 12786 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "c7fc5683934e4c9f885de04addaa2076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0febd027bb4eba809921f54e8e8a4c" I0517 06:45:17.038658 12537 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "c7fc5683934e4c9f885de04addaa2076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7859ab255d4820bf262d97c42cdd80" I0517 06:45:17.039021 12786 raft_consensus.cc:2180] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 LEADER]: Leader election vote request: Denying vote to candidate c7fc5683934e4c9f885de04addaa2076 for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:45:17.039176 12537 raft_consensus.cc:2149] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c7fc5683934e4c9f885de04addaa2076 in current term 1: Already voted for candidate 0d0febd027bb4eba809921f54e8e8a4c in this term. I0517 06:45:17.039973 12596 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 election: Vote denied by peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391). Message: Invalid argument: T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 LEADER]: Leader election vote request: Denying vote to candidate c7fc5683934e4c9f885de04addaa2076 for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:45:17.040264 12592 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 election: Vote denied by peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751). Message: Invalid argument: T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c7fc5683934e4c9f885de04addaa2076 in current term 1: Already voted for candidate 0d0febd027bb4eba809921f54e8e8a4c in this term. I0517 06:45:17.040607 13141 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 election: Requesting vote from peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957) I0517 06:45:17.041811 12289 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "c7fc5683934e4c9f885de04addaa2076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2afd3be3c104d17802599476b02d509" I0517 06:45:17.042356 12289 raft_consensus.cc:2149] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c7fc5683934e4c9f885de04addaa2076 in current term 1: Already voted for candidate 0d0febd027bb4eba809921f54e8e8a4c in this term. I0517 06:45:17.043299 12592 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 election: Vote denied by peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957). Message: Invalid argument: T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c7fc5683934e4c9f885de04addaa2076 in current term 1: Already voted for candidate 0d0febd027bb4eba809921f54e8e8a4c in this term. I0517 06:45:17.043550 12592 leader_election.cc:270] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I0517 06:45:17.044674 13141 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: Snoozing failure detection for 2.249s (election complete) I0517 06:45:17.045112 13141 raft_consensus.cc:2459] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0517 06:45:17.060554 13146 raft_consensus.cc:435] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:17.060827 13146 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: Snoozing failure detection for 1.810s (starting election) I0517 06:45:17.061275 12909 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "c7fc5683934e4c9f885de04addaa2076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" is_pre_election: true I0517 06:45:17.061962 12910 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "b2afd3be3c104d17802599476b02d509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" is_pre_election: true I0517 06:45:17.062489 12908 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "6f7859ab255d4820bf262d97c42cdd80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" is_pre_election: true I0517 06:45:17.063025 12907 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "c7fc5683934e4c9f885de04addaa2076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" I0517 06:45:17.063434 13146 raft_consensus.cc:457] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:17.064644 12910 raft_consensus.cc:2194] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2afd3be3c104d17802599476b02d509 for term 1 because replica is already servicing an update from a current leader or another vote. I0517 06:45:17.065033 12908 raft_consensus.cc:2194] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f7859ab255d4820bf262d97c42cdd80 for term 1 because replica is already servicing an update from a current leader or another vote. I0517 06:45:17.066781 12220 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961). Message: Service unavailable: T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2afd3be3c104d17802599476b02d509 for term 1 because replica is already servicing an update from a current leader or another vote. I0517 06:45:17.067010 12220 leader_election.cc:270] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0517 06:45:17.067029 12468 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961). Message: Service unavailable: T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f7859ab255d4820bf262d97c42cdd80 for term 1 because replica is already servicing an update from a current leader or another vote. I0517 06:45:17.067872 12909 raft_consensus.cc:2149] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c7fc5683934e4c9f885de04addaa2076 in current term 1: Already voted for candidate 0d0febd027bb4eba809921f54e8e8a4c in this term. I0517 06:45:17.068176 12907 raft_consensus.cc:2194] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c7fc5683934e4c9f885de04addaa2076 for term 1 because replica is already servicing an update from a current leader or another vote. I0517 06:45:17.068353 13149 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Snoozing failure detection for 1.979s (election complete) I0517 06:45:17.068574 13149 raft_consensus.cc:2459] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0517 06:45:17.064512 13146 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583) I0517 06:45:17.071012 13146 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391) I0517 06:45:17.076067 13146 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751) I0517 06:45:17.077890 12593 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961). Message: Invalid argument: T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c7fc5683934e4c9f885de04addaa2076 in current term 1: Already voted for candidate 0d0febd027bb4eba809921f54e8e8a4c in this term. I0517 06:45:17.078560 12593 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [CANDIDATE]: Term 1 election: Vote denied by peer 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961). Message: Service unavailable: T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c7fc5683934e4c9f885de04addaa2076 for term 1 because replica is already servicing an update from a current leader or another vote. I0517 06:45:17.077996 13146 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957) I0517 06:45:17.121570 12661 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fc5683934e4c9f885de04addaa2076" is_pre_election: true I0517 06:45:17.121968 12661 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: Snoozing failure detection for 3.272s (vote granted) I0517 06:45:17.122126 12661 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d6b0fe0b45450a809f4a25eb6bae51 in term 1. I0517 06:45:17.123332 12840 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 pre-election: Vote granted by peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583) I0517 06:45:17.131491 12537 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7859ab255d4820bf262d97c42cdd80" is_pre_election: true I0517 06:45:17.131935 12537 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Snoozing failure detection for 2.242s (vote granted) I0517 06:45:17.132124 12537 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d6b0fe0b45450a809f4a25eb6bae51 in term 1. I0517 06:45:17.132899 12840 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751) I0517 06:45:17.133134 12840 leader_election.cc:270] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0517 06:45:17.133615 13146 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: Snoozing failure detection for 2.172s (election complete) I0517 06:45:17.133821 13146 raft_consensus.cc:2511] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0517 06:45:17.134021 13146 raft_consensus.cc:435] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:45:17.134182 13146 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: Snoozing failure detection for 1.931s (starting election) I0517 06:45:17.134346 13146 raft_consensus.cc:2756] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:45:17.139086 13146 raft_consensus.cc:457] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:17.139899 13146 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 election: Requesting vote from peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583) I0517 06:45:17.140220 13146 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 election: Requesting vote from peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391) I0517 06:45:17.140480 13146 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 election: Requesting vote from peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751) I0517 06:45:17.140828 13146 leader_election.cc:243] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 election: Requesting vote from peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957) I0517 06:45:17.141175 12661 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fc5683934e4c9f885de04addaa2076" I0517 06:45:17.141639 12661 raft_consensus.cc:2756] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:45:17.141854 12661 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076: Snoozing failure detection for 2.523s (vote granted) I0517 06:45:17.146701 12661 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d6b0fe0b45450a809f4a25eb6bae51 in term 2. I0517 06:45:17.147945 12537 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7859ab255d4820bf262d97c42cdd80" I0517 06:45:17.148332 12537 raft_consensus.cc:2756] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:45:17.148535 12537 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80: Snoozing failure detection for 2.380s (vote granted) I0517 06:45:17.153221 12537 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d6b0fe0b45450a809f4a25eb6bae51 in term 2. I0517 06:45:17.154145 12840 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 election: Vote granted by peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583) I0517 06:45:17.154858 12840 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 election: Vote granted by peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751) I0517 06:45:17.155098 12840 leader_election.cc:270] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0517 06:45:17.155488 13146 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51: Snoozing failure detection for 2.175s (election complete) I0517 06:45:17.155720 13146 raft_consensus.cc:2511] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 2 FOLLOWER]: Leader election won for term 2 I0517 06:45:17.158252 12785 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0febd027bb4eba809921f54e8e8a4c" I0517 06:45:17.158813 12785 raft_consensus.cc:2180] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 LEADER]: Leader election vote request: Denying vote to candidate 71d6b0fe0b45450a809f4a25eb6bae51 for term 2 because replica is either leader or believes a valid leader to be alive. I0517 06:45:17.159327 12786 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d0febd027bb4eba809921f54e8e8a4c" is_pre_election: true I0517 06:45:17.159828 12786 raft_consensus.cc:2180] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 71d6b0fe0b45450a809f4a25eb6bae51 for term 2 because replica is either leader or believes a valid leader to be alive. I0517 06:45:17.162078 12844 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 election: Vote denied by peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391). Message: Invalid argument: T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 LEADER]: Leader election vote request: Denying vote to candidate 71d6b0fe0b45450a809f4a25eb6bae51 for term 2 because replica is either leader or believes a valid leader to be alive. I0517 06:45:17.162611 12844 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391). Message: Invalid argument: T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 71d6b0fe0b45450a809f4a25eb6bae51 for term 2 because replica is either leader or believes a valid leader to be alive. I0517 06:45:17.162688 12289 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2afd3be3c104d17802599476b02d509" is_pre_election: true I0517 06:45:17.163040 12288 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2344f0911d62405895a6c44edca703bc" candidate_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2afd3be3c104d17802599476b02d509" I0517 06:45:17.163168 12289 raft_consensus.cc:2704] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Snoozing failure detection for 1.889s (vote granted) I0517 06:45:17.163403 12289 raft_consensus.cc:2219] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d6b0fe0b45450a809f4a25eb6bae51 in term 1. I0517 06:45:17.163962 12288 raft_consensus.cc:2194] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 71d6b0fe0b45450a809f4a25eb6bae51 for term 2 because replica is already servicing an update from a current leader or another vote. I0517 06:45:17.163938 13146 raft_consensus.cc:571] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [term 2 LEADER]: Becoming Leader. State: Replica: 71d6b0fe0b45450a809f4a25eb6bae51, State: Running, Role: LEADER I0517 06:45:17.164680 12840 leader_election.cc:379] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 pre-election: Vote granted by peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957) I0517 06:45:17.164422 13146 consensus_queue.cc:204] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:17.165565 12840 leader_election.cc:393] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [CANDIDATE]: Term 2 election: Vote denied by peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957). Message: Service unavailable: T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 71d6b0fe0b45450a809f4a25eb6bae51 for term 2 because replica is already servicing an update from a current leader or another vote. I0517 06:45:17.172518 13077 heartbeater.cc:467] Master 127.4.96.62:50584 was elected leader, sending a full tablet report... I0517 06:45:17.173985 11916 catalog_manager.cc:3686] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 reported cstate change: term changed from 1 to 2, leader changed from 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7) to 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8). New cstate: current_term: 2 leader_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } health_report { overall_health: UNKNOWN } } } I0517 06:45:17.188357 4480 external_mini_cluster.cc:482] 9 TS(s) registered with all masters I0517 06:45:17.191519 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3c4ca03e934448e789a3413ea92ab6c6 to finish bootstrapping I0517 06:45:17.210310 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b63d0bdace5e4d598392de0641274b6c to finish bootstrapping I0517 06:45:17.223626 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b2afd3be3c104d17802599476b02d509 to finish bootstrapping I0517 06:45:17.234329 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver bc2cfadffd3547ae88f42510ed8395f3 to finish bootstrapping I0517 06:45:17.245841 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6f7859ab255d4820bf262d97c42cdd80 to finish bootstrapping I0517 06:45:17.254297 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c7fc5683934e4c9f885de04addaa2076 to finish bootstrapping I0517 06:45:17.265033 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0d0febd027bb4eba809921f54e8e8a4c to finish bootstrapping I0517 06:45:17.274096 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 71d6b0fe0b45450a809f4a25eb6bae51 to finish bootstrapping I0517 06:45:17.283661 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f46a2a66ce864668aa605848aea6f97e to finish bootstrapping I0517 06:45:17.294615 4480 kudu-admin-test.cc:1055] Waiting for Master to see the current replicas... I0517 06:45:17.296952 4480 kudu-admin-test.cc:1062] Tablet locations: tablet_id: "2344f0911d62405895a6c44edca703bc" replicas { ts_info { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" rpc_addresses { host: "127.4.96.8" port: 50961 } } role: LEADER } replicas { ts_info { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" rpc_addresses { host: "127.4.96.6" port: 43583 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" rpc_addresses { host: "127.4.96.7" port: 51391 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" rpc_addresses { host: "127.4.96.5" port: 59751 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "b2afd3be3c104d17802599476b02d509" rpc_addresses { host: "127.4.96.3" port: 49957 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0517 06:45:17.457243 12288 raft_consensus.cc:1057] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 1 FOLLOWER]: Refusing update from remote peer 0d0febd027bb4eba809921f54e8e8a4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:45:17.458405 13118 consensus_queue.cc:963] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:17.474588 12537 raft_consensus.cc:1025] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 2 FOLLOWER]: Rejecting Update request from peer 0d0febd027bb4eba809921f54e8e8a4c for earlier term 1. Current term is 2. Ops: [] I0517 06:45:17.475546 13118 consensus_queue.cc:974] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:17.476675 13118 raft_consensus.cc:2751] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 LEADER]: Stepping down as leader of term 1 I0517 06:45:17.476824 13118 raft_consensus.cc:604] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d0febd027bb4eba809921f54e8e8a4c, State: Running, Role: LEADER I0517 06:45:17.477159 13118 consensus_queue.cc:227] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d6b0fe0b45450a809f4a25eb6bae51" member_type: VOTER last_known_addr { host: "127.4.96.8" port: 50961 } } peers { permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 } } peers { permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 } } peers { permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 } } peers { permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 } } I0517 06:45:17.477736 13118 consensus_peers.cc:456] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c -> Peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751): Closing peer: 6f7859ab255d4820bf262d97c42cdd80 I0517 06:45:17.477900 13118 consensus_peers.cc:456] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c -> Peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583): Closing peer: c7fc5683934e4c9f885de04addaa2076 I0517 06:45:17.478039 13118 consensus_peers.cc:456] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c -> Peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957): Closing peer: b2afd3be3c104d17802599476b02d509 I0517 06:45:17.478166 13118 consensus_peers.cc:456] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c -> Peer 71d6b0fe0b45450a809f4a25eb6bae51 (127.4.96.8:50961): Closing peer: 71d6b0fe0b45450a809f4a25eb6bae51 I0517 06:45:17.478543 13118 raft_consensus.cc:2756] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 1 FOLLOWER]: Advancing to term 2 I0517 06:45:17.652545 12537 raft_consensus.cc:1057] T 2344f0911d62405895a6c44edca703bc P 6f7859ab255d4820bf262d97c42cdd80 [term 2 FOLLOWER]: Refusing update from remote peer 71d6b0fe0b45450a809f4a25eb6bae51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I0517 06:45:17.653472 13146 consensus_queue.cc:963] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f7859ab255d4820bf262d97c42cdd80" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:17.756397 12786 raft_consensus.cc:1057] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 2 FOLLOWER]: Refusing update from remote peer 71d6b0fe0b45450a809f4a25eb6bae51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch) I0517 06:45:17.756553 12786 pending_rounds.cc:76] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Aborting all transactions after (but not including) 0 I0517 06:45:17.756669 12786 pending_rounds.cc:97] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c: Aborting uncommitted NO_OP operation due to leader change: 1.1 I0517 06:45:17.756796 12786 raft_consensus.cc:2594] T 2344f0911d62405895a6c44edca703bc P 0d0febd027bb4eba809921f54e8e8a4c [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader I0517 06:45:17.757771 13146 consensus_queue.cc:963] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d0febd027bb4eba809921f54e8e8a4c" member_type: VOTER last_known_addr { host: "127.4.96.7" port: 51391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:17.764262 12288 raft_consensus.cc:2756] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:45:17.771793 12661 raft_consensus.cc:1057] T 2344f0911d62405895a6c44edca703bc P c7fc5683934e4c9f885de04addaa2076 [term 2 FOLLOWER]: Refusing update from remote peer 71d6b0fe0b45450a809f4a25eb6bae51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I0517 06:45:17.773052 13146 consensus_queue.cc:963] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7fc5683934e4c9f885de04addaa2076" member_type: VOTER last_known_addr { host: "127.4.96.6" port: 43583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:17.776613 12288 raft_consensus.cc:1057] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 2 FOLLOWER]: Refusing update from remote peer 71d6b0fe0b45450a809f4a25eb6bae51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch) I0517 06:45:17.776789 12288 pending_rounds.cc:76] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Aborting all transactions after (but not including) 0 I0517 06:45:17.776899 12288 pending_rounds.cc:97] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509: Aborting uncommitted NO_OP operation due to leader change: 1.1 I0517 06:45:17.777036 12288 raft_consensus.cc:2594] T 2344f0911d62405895a6c44edca703bc P b2afd3be3c104d17802599476b02d509 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader I0517 06:45:17.778257 13167 consensus_queue.cc:963] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2afd3be3c104d17802599476b02d509" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 49957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:17.792289 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 12459 W0517 06:45:17.812429 12592 connection.cc:512] client connection to 127.4.96.5:59751 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:59751): Connection reset by peer (error 104) W0517 06:45:17.812456 11896 connection.cc:512] client connection to 127.4.96.5:59751 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:59751): Connection reset by peer (error 104) W0517 06:45:17.813369 11900 connection.cc:512] server connection from 127.4.96.5:39442 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:39442): Connection reset by peer (error 104) W0517 06:45:17.813477 12219 connection.cc:512] client connection to 127.4.96.5:59751 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:59751): Connection reset by peer (error 104) W0517 06:45:17.813521 12841 connection.cc:512] server connection from 127.4.96.5:50488 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:50488): Connection reset by peer (error 104) W0517 06:45:17.814373 12596 connection.cc:512] server connection from 127.4.96.5:39669 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:39669): Connection reset by peer (error 104) W0517 06:45:17.814481 12720 connection.cc:512] server connection from 127.4.96.5:40578 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:40578): Connection reset by peer (error 104) W0517 06:45:17.815348 12716 connection.cc:512] client connection to 127.4.96.5:59751 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:59751): Connection reset by peer (error 104) W0517 06:45:17.815373 12840 connection.cc:512] client connection to 127.4.96.5:59751 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:59751): Connection reset by peer (error 104) W0517 06:45:17.815433 12221 connection.cc:512] server connection from 127.4.96.5:55186 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:55186): Connection reset by peer (error 104) W0517 06:45:17.816223 11887 connection.cc:512] client connection to 127.4.96.5:59751 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:59751): Connection reset by peer (error 104) W0517 06:45:17.817004 11883 connection.cc:512] client connection to 127.4.96.5:59751 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:59751): Connection reset by peer (error 104) I0517 06:45:17.817162 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 12583 W0517 06:45:17.841164 11898 connection.cc:512] server connection from 127.4.96.6:56588 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.6:56588): Connection reset by peer (error 104) W0517 06:45:17.841164 12840 connection.cc:512] client connection to 127.4.96.6:43583 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.6:43583): Connection reset by peer (error 104) W0517 06:45:17.841265 12716 connection.cc:512] client connection to 127.4.96.6:43583 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.6:43583): Connection reset by peer (error 104) W0517 06:45:17.841722 11896 connection.cc:512] client connection to 127.4.96.6:43583 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.6:43583): Connection reset by peer (error 104) W0517 06:45:17.842066 11883 connection.cc:512] client connection to 127.4.96.6:43583 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.6:43583): Connection reset by peer (error 104) W0517 06:45:17.846135 12219 connection.cc:512] client connection to 127.4.96.6:43583 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.6:43583): Connection reset by peer (error 104) I0517 06:45:17.846868 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 12211 W0517 06:45:17.848239 11887 connection.cc:512] client connection to 127.4.96.6:43583 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.6:43583): Connection reset by peer (error 104) W0517 06:45:17.849043 12841 connection.cc:512] server connection from 127.4.96.6:43471 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.6:43471): Connection reset by peer (error 104) W0517 06:45:17.849798 12720 connection.cc:512] server connection from 127.4.96.6:35035 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.6:35035): Connection reset by peer (error 104) W0517 06:45:17.869886 12841 connection.cc:512] server connection from 127.4.96.3:45942 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:45942): Connection reset by peer (error 104) W0517 06:45:17.869885 11896 connection.cc:512] client connection to 127.4.96.3:49957 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:49957): Connection reset by peer (error 104) W0517 06:45:17.869887 11898 connection.cc:512] server connection from 127.4.96.3:53598 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:53598): Connection reset by peer (error 104) W0517 06:45:17.870648 11887 connection.cc:512] client connection to 127.4.96.3:49957 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:49957): Connection reset by peer (error 104) W0517 06:45:17.870645 12716 connection.cc:512] client connection to 127.4.96.3:49957 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:49957): Connection reset by peer (error 104) W0517 06:45:17.870923 12840 connection.cc:512] client connection to 127.4.96.3:49957 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:49957): Connection reset by peer (error 104) W0517 06:45:17.871403 11883 connection.cc:512] client connection to 127.4.96.3:49957 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:49957): Connection reset by peer (error 104) I0517 06:45:17.872352 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 12708 W0517 06:45:17.891104 11898 connection.cc:512] server connection from 127.4.96.7:44773 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.7:44773): Connection reset by peer (error 104) W0517 06:45:17.891106 11900 connection.cc:512] client connection to 127.4.96.7:51391 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.7:51391): Connection reset by peer (error 104) W0517 06:45:17.892138 12844 connection.cc:512] server connection from 127.4.96.7:42120 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.7:42120): Connection reset by peer (error 104) W0517 06:45:17.892141 11887 connection.cc:512] client connection to 127.4.96.7:51391 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.7:51391): Connection reset by peer (error 104) W0517 06:45:17.892833 12844 connection.cc:512] client connection to 127.4.96.7:51391 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.7:51391): Connection reset by peer (error 104) W0517 06:45:17.893034 11886 connection.cc:512] client connection to 127.4.96.7:51391 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.7:51391): Connection reset by peer (error 104) I0517 06:45:17.893878 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 11888 W0517 06:45:17.918979 12840 connection.cc:512] client connection to 127.4.96.62:50584 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50584): Connection reset by peer (error 104) W0517 06:45:17.919095 11967 connection.cc:512] client connection to 127.4.96.62:50584 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50584): Connection reset by peer (error 104) W0517 06:45:17.919118 12343 connection.cc:512] client connection to 127.4.96.62:50584 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50584): Connection reset by peer (error 104) W0517 06:45:17.919839 12840 connection.cc:512] server connection from 127.0.0.1:53293 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:53293): Connection reset by peer (error 104) W0517 06:45:17.920199 11887 connection.cc:512] client connection to 127.4.96.62:50584 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50584): Connection reset by peer (error 104) I0517 06:45:17.920686 4480 kudu-admin-test.cc:1079] Forcing unsafe config change on tserver c7fc5683934e4c9f885de04addaa2076 W0517 06:45:17.926406 13082 connection.cc:512] client connection to 127.4.96.62:50584 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50584): Connection reset by peer (error 104) W0517 06:45:17.927664 12964 connection.cc:512] client connection to 127.4.96.62:50584 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50584): Connection reset by peer (error 104) W0517 06:45:17.928517 12094 connection.cc:512] client connection to 127.4.96.62:50584 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50584): Connection reset by peer (error 104) W0517 06:45:18.035997 13171 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:18.090936 12456 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:50584: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.96.62:50584: connect: Connection refused (error 111) W0517 06:45:18.174983 12207 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:50584: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.96.62:50584: connect: Connection refused (error 111) W0517 06:45:18.177084 13077 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:50584: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.96.62:50584: connect: Connection refused (error 111) W0517 06:45:18.201035 12840 consensus_peers.cc:434] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 -> Peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583): Couldn't send request to peer c7fc5683934e4c9f885de04addaa2076 for tablet 2344f0911d62405895a6c44edca703bc. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.6:43583: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0517 06:45:18.201652 12953 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:50584: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.96.62:50584: connect: Connection refused (error 111) Network error: Client connection negotiation failed: client connection to 127.4.96.7:51391: connect: Connection refused (error 111) W0517 06:45:18.234182 12840 consensus_peers.cc:434] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 -> Peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751): Couldn't send request to peer 6f7859ab255d4820bf262d97c42cdd80 for tablet 2344f0911d62405895a6c44edca703bc. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:59751: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0517 06:45:18.317276 12844 consensus_peers.cc:434] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 -> Peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391): Couldn't send request to peer 0d0febd027bb4eba809921f54e8e8a4c for tablet 2344f0911d62405895a6c44edca703bc. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.7:51391: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0517 06:45:18.347021 12840 consensus_peers.cc:434] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 -> Peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957): Couldn't send request to peer b2afd3be3c104d17802599476b02d509 for tablet 2344f0911d62405895a6c44edca703bc. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.3:49957: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0517 06:45:18.347472 12953 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:50584: Network error: Failed to ping master at 127.4.96.62:50584: Client connection negotiation failed: client connection to 127.4.96.62:50584: connect: Connection refused (error 111) W0517 06:45:18.590986 12840 consensus_peers.cc:434] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 -> Peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583): Couldn't send request to peer c7fc5683934e4c9f885de04addaa2076 for tablet 2344f0911d62405895a6c44edca703bc. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.6:43583: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0517 06:45:18.591454 12953 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:50584: Network error: Failed to ping master at 127.4.96.62:50584: Client connection negotiation failed: client connection to 127.4.96.62:50584: connect: Connection refused (error 111) W0517 06:45:18.651249 12082 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:50584: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.96.62:50584: connect: Connection refused (error 111) W0517 06:45:18.724278 12840 consensus_peers.cc:434] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 -> Peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957): Couldn't send request to peer b2afd3be3c104d17802599476b02d509 for tablet 2344f0911d62405895a6c44edca703bc. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.3:49957: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0517 06:45:18.724675 12953 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:50584: Network error: Failed to ping master at 127.4.96.62:50584: Client connection negotiation failed: client connection to 127.4.96.62:50584: connect: Connection refused (error 111) W0517 06:45:18.790195 12840 consensus_peers.cc:434] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 -> Peer 6f7859ab255d4820bf262d97c42cdd80 (127.4.96.5:59751): Couldn't send request to peer 6f7859ab255d4820bf262d97c42cdd80 for tablet 2344f0911d62405895a6c44edca703bc. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:59751: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0517 06:45:18.790664 12953 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:50584: Network error: Failed to ping master at 127.4.96.62:50584: Client connection negotiation failed: client connection to 127.4.96.62:50584: connect: Connection refused (error 111) W0517 06:45:18.854002 12844 consensus_peers.cc:434] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 -> Peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391): Couldn't send request to peer 0d0febd027bb4eba809921f54e8e8a4c for tablet 2344f0911d62405895a6c44edca703bc. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.7:51391: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0517 06:45:18.854220 12953 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:50584: Network error: Failed to ping master at 127.4.96.62:50584: Client connection negotiation failed: client connection to 127.4.96.62:50584: connect: Connection refused (error 111) W0517 06:45:19.073371 12840 consensus_peers.cc:434] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 -> Peer c7fc5683934e4c9f885de04addaa2076 (127.4.96.6:43583): Couldn't send request to peer c7fc5683934e4c9f885de04addaa2076 for tablet 2344f0911d62405895a6c44edca703bc. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.6:43583: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0517 06:45:19.093060 12456 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:50584: Network error: Failed to ping master at 127.4.96.62:50584: Client connection negotiation failed: client connection to 127.4.96.62:50584: connect: Connection refused (error 111) W0517 06:45:19.177320 12207 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:50584: Network error: Failed to ping master at 127.4.96.62:50584: Client connection negotiation failed: client connection to 127.4.96.62:50584: connect: Connection refused (error 111) W0517 06:45:19.179183 13077 heartbeater.cc:587] Failed to heartbeat to 127.4.96.62:50584: Network error: Failed to ping master at 127.4.96.62:50584: Client connection negotiation failed: client connection to 127.4.96.62:50584: connect: Connection refused (error 111) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/tools/kudu-admin-test.cc:1096: Failure Failed Bad status: Runtime error: /tmp/dist-test-task4QLboq/build/tsan/bin/kudu: process exited with non-zero status 1 I0517 06:45:19.219760 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 11950 W0517 06:45:19.241495 11887 connection.cc:512] client connection to 127.4.96.1:38160 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:38160): Connection reset by peer (error 104) I0517 06:45:19.241645 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 12086 I0517 06:45:19.266521 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 12334 W0517 06:45:19.267532 11887 connection.cc:512] client connection to 127.4.96.2:51044 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:51044): Connection reset by peer (error 104) W0517 06:45:19.285991 12840 consensus_peers.cc:434] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 -> Peer b2afd3be3c104d17802599476b02d509 (127.4.96.3:49957): Couldn't send request to peer b2afd3be3c104d17802599476b02d509 for tablet 2344f0911d62405895a6c44edca703bc. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.3:49957: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0517 06:45:19.286134 12844 consensus_peers.cc:434] T 2344f0911d62405895a6c44edca703bc P 71d6b0fe0b45450a809f4a25eb6bae51 -> Peer 0d0febd027bb4eba809921f54e8e8a4c (127.4.96.7:51391): Couldn't send request to peer 0d0febd027bb4eba809921f54e8e8a4c for tablet 2344f0911d62405895a6c44edca703bc. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.7:51391: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0517 06:45:19.290012 11887 connection.cc:512] client connection to 127.4.96.4:39443 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:39443): Connection reset by peer (error 104) I0517 06:45:19.294095 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 12832 W0517 06:45:19.313988 11887 connection.cc:512] client connection to 127.4.96.8:50961 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.8:50961): Connection reset by peer (error 104) W0517 06:45:19.313993 11884 connection.cc:512] client connection to 127.4.96.8:50961 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.8:50961): Connection reset by peer (error 104) I0517 06:45:19.315212 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 12956 W0517 06:45:19.339079 11887 connection.cc:512] client connection to 127.4.96.9:37079 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.9:37079): Connection reset by peer (error 104) I0517 06:45:19.348232 4480 test_util.cc:123] ----------------------------------------------- I0517 06:45:19.348376 4480 test_util.cc:124] Had fatal failures, leaving test files at /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1526539434354140-4480 [ FAILED ] AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs (7774 ms) [ RUN ] AdminCliTest.TestLeaderStepDown I0517 06:45:19.350246 4480 test_util.cc:200] Using random seed: 768169430 I0517 06:45:19.353252 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:45:19.353355 4480 ts_itest-base.cc:114] -------------- I0517 06:45:19.353446 4480 ts_itest-base.cc:115] 3 tablet servers I0517 06:45:19.353531 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:45:19.353615 4480 ts_itest-base.cc:117] -------------- I0517 06:45:19.355952 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:34571 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:19.469384 13194 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:19.469712 13194 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:19.478691 13194 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:45:19.478821 13194 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:19.478936 13194 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:45:19.479070 13194 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:45:19.492691 13194 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:34571 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:19.500321 13194 master_main.cc:80] Initializing master server... I0517 06:45:19.501826 13194 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 158819us I0517 06:45:19.502552 13194 fs_manager.cc:260] Metadata directory not provided I0517 06:45:19.502682 13194 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:19.502899 13194 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:45:19.503015 13194 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:19.510192 13194 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "44b6f56abe314380b8bad4d04c945e80" format_stamp: "Formatted at 2018-05-17 06:45:19 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:19.510864 13194 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "44b6f56abe314380b8bad4d04c945e80" format_stamp: "Formatted at 2018-05-17 06:45:19 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:19.520012 13194 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:45:19.521128 13194 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:19.521548 13194 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:19.527940 13194 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:45:19.528051 13194 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "44b6f56abe314380b8bad4d04c945e80" format_stamp: "Formatted at 2018-05-17 06:45:19 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:19.528254 13194 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:19.696431 13194 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:19.698151 13194 master_main.cc:83] Starting Master server... I0517 06:45:19.737884 13194 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:34571 I0517 06:45:19.738767 13194 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:45:19.738876 13194 webserver.cc:184] Document root disabled I0517 06:45:19.740074 13194 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:52749/ I0517 06:45:19.740809 13194 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:45:19.743449 13255 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:19.748927 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 13194 I0517 06:45:19.750387 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:34571 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:19.759445 13255 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80: Bootstrap starting. I0517 06:45:19.761616 13255 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80: No blocks or log segments found. Creating new log. I0517 06:45:19.762805 13255 log.cc:520] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80: Log is configured to *not* fsync() on all Append() calls I0517 06:45:19.765110 13255 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80: No bootstrap required, opened a new log I0517 06:45:19.776790 13255 raft_consensus.cc:303] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b6f56abe314380b8bad4d04c945e80" member_type: VOTER } I0517 06:45:19.777087 13255 raft_consensus.cc:329] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:19.777209 13255 raft_consensus.cc:604] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b6f56abe314380b8bad4d04c945e80, State: Initialized, Role: FOLLOWER I0517 06:45:19.777653 13255 consensus_queue.cc:227] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b6f56abe314380b8bad4d04c945e80" member_type: VOTER } I0517 06:45:19.778000 13255 raft_consensus.cc:343] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:45:19.778142 13255 raft_consensus.cc:435] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:45:19.778285 13255 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80: Snoozing failure detection for 2.070s (starting election) I0517 06:45:19.778424 13255 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:19.780779 13255 raft_consensus.cc:457] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b6f56abe314380b8bad4d04c945e80" member_type: VOTER } I0517 06:45:19.781175 13255 leader_election.cc:270] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:19.782742 13258 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80: Snoozing failure detection for 2.102s (election complete) I0517 06:45:19.783038 13258 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:19.794445 13258 raft_consensus.cc:571] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [term 1 LEADER]: Becoming Leader. State: Replica: 44b6f56abe314380b8bad4d04c945e80, State: Running, Role: LEADER I0517 06:45:19.794833 13258 consensus_queue.cc:204] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b6f56abe314380b8bad4d04c945e80" member_type: VOTER } I0517 06:45:19.798172 13259 sys_catalog.cc:335] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44b6f56abe314380b8bad4d04c945e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b6f56abe314380b8bad4d04c945e80" member_type: VOTER } } I0517 06:45:19.798728 13259 sys_catalog.cc:338] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [sys.catalog]: This master's current role is: LEADER I0517 06:45:19.798506 13260 sys_catalog.cc:335] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44b6f56abe314380b8bad4d04c945e80. Latest consensus state: current_term: 1 leader_uuid: "44b6f56abe314380b8bad4d04c945e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b6f56abe314380b8bad4d04c945e80" member_type: VOTER } } I0517 06:45:19.857801 13260 sys_catalog.cc:338] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [sys.catalog]: This master's current role is: LEADER I0517 06:45:19.798712 13255 sys_catalog.cc:335] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "44b6f56abe314380b8bad4d04c945e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b6f56abe314380b8bad4d04c945e80" member_type: VOTER } } I0517 06:45:19.860800 13255 sys_catalog.cc:338] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [sys.catalog]: This master's current role is: LEADER I0517 06:45:19.861974 13255 sys_catalog.cc:422] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:45:19.862200 13262 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:45:19.863677 13194 master_main.cc:86] Master server successfully started. I0517 06:45:19.866950 13262 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:45:19.903862 13262 catalog_manager.cc:899] Generated new certificate authority record I0517 06:45:19.905359 13262 catalog_manager.cc:1021] Loading token signing keys... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:19.905819 13256 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:19.906312 13256 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:19.920549 13256 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 I0517 06:45:19.921049 13262 catalog_manager.cc:3962] T 00000000000000000000000000000000 P 44b6f56abe314380b8bad4d04c945e80: Generated new TSK 0 W0517 06:45:19.921090 13256 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:45:19.941336 13256 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:34571 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:19.954481 13256 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:19.956948 13256 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 158819us I0517 06:45:19.957526 13256 fs_manager.cc:260] Metadata directory not provided I0517 06:45:19.957702 13256 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:19.957995 13256 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:45:19.958158 13256 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:19.970865 13256 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "95463d19beac48dda9314dd2e97c2774" format_stamp: "Formatted at 2018-05-17 06:45:19 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:19.971786 13256 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "95463d19beac48dda9314dd2e97c2774" format_stamp: "Formatted at 2018-05-17 06:45:19 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:19.985909 13256 fs_manager.cc:495] Time spent creating directory manager: real 0.014s user 0.016s sys 0.000s I0517 06:45:19.987556 13256 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:19.988101 13256 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:19.998482 13256 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0517 06:45:19.998636 13256 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "95463d19beac48dda9314dd2e97c2774" format_stamp: "Formatted at 2018-05-17 06:45:19 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:19.998929 13256 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:20.157824 13256 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:20.160981 13256 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:20.162137 13256 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:20.315122 13256 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:48936 I0517 06:45:20.316473 13256 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:45:20.316617 13256 webserver.cc:184] Document root disabled I0517 06:45:20.318109 13256 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:49847/ I0517 06:45:20.318984 13256 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:45:20.327836 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 13256 I0517 06:45:20.329102 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:34571 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:20.330781 13256 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:20.372575 13387 heartbeater.cc:346] Connected to a master server at 127.4.96.62:34571 I0517 06:45:20.372817 13387 heartbeater.cc:426] Registering TS with master... I0517 06:45:20.373353 13387 heartbeater.cc:475] Master 127.4.96.62:34571 requested a full tablet report, sending... I0517 06:45:20.375288 13222 ts_manager.cc:81] Registered new tserver with Master: 95463d19beac48dda9314dd2e97c2774 (127.4.96.1:48936) I0517 06:45:20.377801 13222 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:51926 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:20.464550 13391 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:20.464866 13391 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:20.472796 13391 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:20.473109 13391 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:45:20.488518 13391 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:34571 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:20.496538 13391 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:20.498350 13391 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 159319us I0517 06:45:20.498857 13391 fs_manager.cc:260] Metadata directory not provided I0517 06:45:20.498986 13391 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:20.499162 13391 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:45:20.499264 13391 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:20.506690 13391 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "e7a082460a324e97b17b1e7660ab3e7b" format_stamp: "Formatted at 2018-05-17 06:45:20 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:20.507341 13391 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "e7a082460a324e97b17b1e7660ab3e7b" format_stamp: "Formatted at 2018-05-17 06:45:20 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:20.531229 13391 fs_manager.cc:495] Time spent creating directory manager: real 0.024s user 0.016s sys 0.008s I0517 06:45:20.532331 13391 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:20.532713 13391 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:20.547116 13391 fs_manager.cc:417] Time spent opening block manager: real 0.010s user 0.004s sys 0.008s I0517 06:45:20.547240 13391 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "e7a082460a324e97b17b1e7660ab3e7b" format_stamp: "Formatted at 2018-05-17 06:45:20 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:20.547441 13391 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:20.791959 13391 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:20.794709 13391 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:20.806237 13391 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:21.202955 13391 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:56429 I0517 06:45:21.203827 13391 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:45:21.203948 13391 webserver.cc:184] Document root disabled I0517 06:45:21.205139 13391 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:36529/ I0517 06:45:21.205802 13391 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:45:21.212399 13391 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:21.214303 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 13391 I0517 06:45:21.215216 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:34571 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:21.227537 13512 heartbeater.cc:346] Connected to a master server at 127.4.96.62:34571 I0517 06:45:21.227763 13512 heartbeater.cc:426] Registering TS with master... I0517 06:45:21.228164 13512 heartbeater.cc:475] Master 127.4.96.62:34571 requested a full tablet report, sending... I0517 06:45:21.229571 13222 ts_manager.cc:81] Registered new tserver with Master: e7a082460a324e97b17b1e7660ab3e7b (127.4.96.2:56429) I0517 06:45:21.231216 13222 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:39235 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:21.321158 13516 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:21.321470 13516 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:21.329291 13516 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:21.329605 13516 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:45:21.342087 13516 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:34571 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:21.349918 13516 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:21.351703 13516 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 159819us I0517 06:45:21.352105 13516 fs_manager.cc:260] Metadata directory not provided I0517 06:45:21.352213 13516 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:21.352389 13516 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:45:21.352489 13516 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:21.359922 13516 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "4b0d7495352c44fe87e7c441ff955e98" format_stamp: "Formatted at 2018-05-17 06:45:21 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:21.360574 13516 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "4b0d7495352c44fe87e7c441ff955e98" format_stamp: "Formatted at 2018-05-17 06:45:21 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:21.369784 13516 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:45:21.370882 13516 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:21.371256 13516 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:21.377486 13516 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:45:21.377593 13516 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "4b0d7495352c44fe87e7c441ff955e98" format_stamp: "Formatted at 2018-05-17 06:45:21 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:21.377805 13516 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:21.380906 13387 heartbeater.cc:467] Master 127.4.96.62:34571 was elected leader, sending a full tablet report... I0517 06:45:21.584971 13516 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:21.587291 13516 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:21.588210 13516 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:21.672835 13516 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:59847 I0517 06:45:21.673707 13516 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:45:21.673802 13516 webserver.cc:184] Document root disabled I0517 06:45:21.674979 13516 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:50097/ I0517 06:45:21.675639 13516 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:45:21.680423 13516 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:21.681601 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 13516 I0517 06:45:21.697244 13637 heartbeater.cc:346] Connected to a master server at 127.4.96.62:34571 I0517 06:45:21.697530 13637 heartbeater.cc:426] Registering TS with master... I0517 06:45:21.698113 13637 heartbeater.cc:475] Master 127.4.96.62:34571 requested a full tablet report, sending... I0517 06:45:21.699652 13222 ts_manager.cc:81] Registered new tserver with Master: 4b0d7495352c44fe87e7c441ff955e98 (127.4.96.3:59847) I0517 06:45:21.700927 13222 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:34533 I0517 06:45:21.702673 4480 external_mini_cluster.cc:482] 3 TS(s) registered with all masters I0517 06:45:21.720470 13221 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50040: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0517 06:45:21.728960 13221 catalog_manager.cc:1463] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0517 06:45:21.758041 13324 tablet_service.cc:759] Processing CreateTablet for tablet 69e4a297899a4d6785d0745bbe5a3ffd (table=TestTable [id=de8c7b2085ff428995c6b96e30da116c]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:21.758857 13324 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 69e4a297899a4d6785d0745bbe5a3ffd. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:21.766819 13449 tablet_service.cc:759] Processing CreateTablet for tablet 69e4a297899a4d6785d0745bbe5a3ffd (table=TestTable [id=de8c7b2085ff428995c6b96e30da116c]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:21.767460 13449 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 69e4a297899a4d6785d0745bbe5a3ffd. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:21.774111 13324 ts_tablet_manager.cc:1173] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:21.775766 13657 ts_tablet_manager.cc:1028] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: Bootstrapping tablet I0517 06:45:21.776257 13657 tablet_bootstrap.cc:436] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: Bootstrap starting. I0517 06:45:21.778226 13574 tablet_service.cc:759] Processing CreateTablet for tablet 69e4a297899a4d6785d0745bbe5a3ffd (table=TestTable [id=de8c7b2085ff428995c6b96e30da116c]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:21.778867 13574 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 69e4a297899a4d6785d0745bbe5a3ffd. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:21.779707 13657 tablet_bootstrap.cc:587] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: No blocks or log segments found. Creating new log. I0517 06:45:21.780756 13449 ts_tablet_manager.cc:1173] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:21.781240 13657 log.cc:520] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: Log is configured to *not* fsync() on all Append() calls I0517 06:45:21.790357 13574 ts_tablet_manager.cc:1173] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:21.794673 13660 ts_tablet_manager.cc:1028] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Bootstrapping tablet I0517 06:45:21.795123 13660 tablet_bootstrap.cc:436] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Bootstrap starting. I0517 06:45:21.797979 13660 tablet_bootstrap.cc:587] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: No blocks or log segments found. Creating new log. I0517 06:45:21.799500 13660 log.cc:520] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Log is configured to *not* fsync() on all Append() calls I0517 06:45:21.801100 13658 ts_tablet_manager.cc:1028] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: Bootstrapping tablet I0517 06:45:21.801491 13658 tablet_bootstrap.cc:436] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: Bootstrap starting. I0517 06:45:21.801582 13657 tablet_bootstrap.cc:436] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: No bootstrap required, opened a new log I0517 06:45:21.801995 13657 ts_tablet_manager.cc:1045] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.004s I0517 06:45:21.802398 13660 tablet_bootstrap.cc:436] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: No bootstrap required, opened a new log I0517 06:45:21.802721 13660 ts_tablet_manager.cc:1045] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s I0517 06:45:21.804347 13658 tablet_bootstrap.cc:587] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: No blocks or log segments found. Creating new log. I0517 06:45:21.805528 13658 log.cc:520] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: Log is configured to *not* fsync() on all Append() calls I0517 06:45:21.817520 13657 raft_consensus.cc:303] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:21.818297 13657 raft_consensus.cc:329] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:21.818493 13657 raft_consensus.cc:604] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95463d19beac48dda9314dd2e97c2774, State: Initialized, Role: FOLLOWER I0517 06:45:21.819095 13657 consensus_queue.cc:227] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:21.820061 13660 raft_consensus.cc:303] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:21.820772 13660 raft_consensus.cc:329] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:21.820948 13660 raft_consensus.cc:604] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0d7495352c44fe87e7c441ff955e98, State: Initialized, Role: FOLLOWER I0517 06:45:21.821496 13660 consensus_queue.cc:227] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:21.823972 13637 heartbeater.cc:467] Master 127.4.96.62:34571 was elected leader, sending a full tablet report... I0517 06:45:21.824535 13658 tablet_bootstrap.cc:436] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: No bootstrap required, opened a new log I0517 06:45:21.824868 13658 ts_tablet_manager.cc:1045] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.016s I0517 06:45:21.826761 13657 ts_tablet_manager.cc:1073] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: Time spent starting tablet: real 0.025s user 0.016s sys 0.004s I0517 06:45:21.834873 13660 ts_tablet_manager.cc:1073] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Time spent starting tablet: real 0.032s user 0.016s sys 0.000s I0517 06:45:21.840185 13658 raft_consensus.cc:303] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:21.840916 13658 raft_consensus.cc:329] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:21.841127 13658 raft_consensus.cc:604] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a082460a324e97b17b1e7660ab3e7b, State: Initialized, Role: FOLLOWER I0517 06:45:21.841632 13658 consensus_queue.cc:227] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:21.843814 13512 heartbeater.cc:467] Master 127.4.96.62:34571 was elected leader, sending a full tablet report... I0517 06:45:21.844959 13658 ts_tablet_manager.cc:1073] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s W0517 06:45:21.831496 13388 tablet.cc:1678] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:45:21.930405 13638 tablet.cc:1678] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:21.985116 13664 raft_consensus.cc:435] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:22.010013 13664 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Snoozing failure detection for 2.148s (starting election) I0517 06:45:22.010219 13664 raft_consensus.cc:457] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:22.011145 13664 leader_election.cc:243] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 95463d19beac48dda9314dd2e97c2774 (127.4.96.1:48936) I0517 06:45:22.011577 13664 leader_election.cc:243] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e7a082460a324e97b17b1e7660ab3e7b (127.4.96.2:56429) W0517 06:45:21.962080 13513 tablet.cc:1678] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:22.023192 13344 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "69e4a297899a4d6785d0745bbe5a3ffd" candidate_uuid: "4b0d7495352c44fe87e7c441ff955e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95463d19beac48dda9314dd2e97c2774" is_pre_election: true I0517 06:45:22.023633 13344 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: Snoozing failure detection for 1.861s (vote granted) I0517 06:45:22.023823 13344 raft_consensus.cc:2219] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0d7495352c44fe87e7c441ff955e98 in term 0. I0517 06:45:22.024756 13524 leader_election.cc:379] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 95463d19beac48dda9314dd2e97c2774 (127.4.96.1:48936) I0517 06:45:22.024940 13524 leader_election.cc:270] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:45:22.025370 13664 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Snoozing failure detection for 1.673s (election complete) I0517 06:45:22.025600 13664 raft_consensus.cc:2511] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:45:22.025790 13664 raft_consensus.cc:435] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:45:22.026011 13664 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Snoozing failure detection for 1.750s (starting election) I0517 06:45:22.026154 13664 raft_consensus.cc:2756] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:22.029356 13664 raft_consensus.cc:457] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:22.030021 13664 leader_election.cc:243] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [CANDIDATE]: Term 1 election: Requesting vote from peer 95463d19beac48dda9314dd2e97c2774 (127.4.96.1:48936) I0517 06:45:22.030303 13664 leader_election.cc:243] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [CANDIDATE]: Term 1 election: Requesting vote from peer e7a082460a324e97b17b1e7660ab3e7b (127.4.96.2:56429) I0517 06:45:22.030966 13344 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "69e4a297899a4d6785d0745bbe5a3ffd" candidate_uuid: "4b0d7495352c44fe87e7c441ff955e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95463d19beac48dda9314dd2e97c2774" I0517 06:45:22.031268 13344 raft_consensus.cc:2756] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:22.031457 13344 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: Snoozing failure detection for 1.868s (vote granted) I0517 06:45:22.031620 13469 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "69e4a297899a4d6785d0745bbe5a3ffd" candidate_uuid: "4b0d7495352c44fe87e7c441ff955e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a082460a324e97b17b1e7660ab3e7b" is_pre_election: true I0517 06:45:22.031945 13468 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "69e4a297899a4d6785d0745bbe5a3ffd" candidate_uuid: "4b0d7495352c44fe87e7c441ff955e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a082460a324e97b17b1e7660ab3e7b" I0517 06:45:22.032217 13469 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: Snoozing failure detection for 1.634s (vote granted) I0517 06:45:22.032452 13469 raft_consensus.cc:2219] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0d7495352c44fe87e7c441ff955e98 in term 0. I0517 06:45:22.032847 13468 raft_consensus.cc:2194] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b0d7495352c44fe87e7c441ff955e98 for term 1 because replica is already servicing an update from a current leader or another vote. I0517 06:45:22.033715 13528 leader_election.cc:379] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [CANDIDATE]: Term 1 pre-election: Vote granted by peer e7a082460a324e97b17b1e7660ab3e7b (127.4.96.2:56429) I0517 06:45:22.034231 13344 raft_consensus.cc:2219] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0d7495352c44fe87e7c441ff955e98 in term 1. I0517 06:45:22.034279 13528 leader_election.cc:393] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [CANDIDATE]: Term 1 election: Vote denied by peer e7a082460a324e97b17b1e7660ab3e7b (127.4.96.2:56429). Message: Service unavailable: T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b0d7495352c44fe87e7c441ff955e98 for term 1 because replica is already servicing an update from a current leader or another vote. I0517 06:45:22.034883 13524 leader_election.cc:379] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [CANDIDATE]: Term 1 election: Vote granted by peer 95463d19beac48dda9314dd2e97c2774 (127.4.96.1:48936) I0517 06:45:22.035053 13524 leader_election.cc:270] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:22.035342 13664 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Snoozing failure detection for 2.201s (election complete) I0517 06:45:22.035490 13664 raft_consensus.cc:2511] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:22.036458 13664 raft_consensus.cc:571] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0d7495352c44fe87e7c441ff955e98, State: Running, Role: LEADER I0517 06:45:22.036926 13664 consensus_queue.cc:204] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:22.042029 13222 catalog_manager.cc:3686] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 reported cstate change: term changed from 0 to 1, leader changed from to 4b0d7495352c44fe87e7c441ff955e98 (127.4.96.3). New cstate: current_term: 1 leader_uuid: "4b0d7495352c44fe87e7c441ff955e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } health_report { overall_health: UNKNOWN } } } I0517 06:45:22.119825 4480 external_mini_cluster.cc:482] 3 TS(s) registered with all masters I0517 06:45:22.121819 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 95463d19beac48dda9314dd2e97c2774 to finish bootstrapping I0517 06:45:22.130527 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e7a082460a324e97b17b1e7660ab3e7b to finish bootstrapping I0517 06:45:22.139474 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4b0d7495352c44fe87e7c441ff955e98 to finish bootstrapping I0517 06:45:22.496256 13344 raft_consensus.cc:1057] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 1 FOLLOWER]: Refusing update from remote peer 4b0d7495352c44fe87e7c441ff955e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:45:22.497388 13664 consensus_queue.cc:963] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:22.500100 13594 tablet_service.cc:1138] Received LeaderStepDown RPC: tablet_id: "69e4a297899a4d6785d0745bbe5a3ffd" dest_uuid: "4b0d7495352c44fe87e7c441ff955e98" from {username='slave'} at 127.0.0.1:45748 I0517 06:45:22.500411 13594 raft_consensus.cc:530] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 1 LEADER]: Received request to step down I0517 06:45:22.500538 13594 raft_consensus.cc:604] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4b0d7495352c44fe87e7c441ff955e98, State: Running, Role: LEADER I0517 06:45:22.500963 13594 consensus_queue.cc:227] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:22.501586 13594 consensus_peers.cc:456] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 -> Peer e7a082460a324e97b17b1e7660ab3e7b (127.4.96.2:56429): Closing peer: e7a082460a324e97b17b1e7660ab3e7b I0517 06:45:22.501762 13594 consensus_peers.cc:456] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 -> Peer 95463d19beac48dda9314dd2e97c2774 (127.4.96.1:48936): Closing peer: 95463d19beac48dda9314dd2e97c2774 I0517 06:45:22.502024 13594 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Snoozing failure detection for 3.000s (explicit stepdown request) I0517 06:45:23.668319 13687 raft_consensus.cc:435] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:23.668514 13687 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: Snoozing failure detection for 2.055s (starting election) I0517 06:45:23.668682 13687 raft_consensus.cc:457] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:23.669483 13687 leader_election.cc:243] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 95463d19beac48dda9314dd2e97c2774 (127.4.96.1:48936) I0517 06:45:23.669864 13687 leader_election.cc:243] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 4b0d7495352c44fe87e7c441ff955e98 (127.4.96.3:59847) I0517 06:45:23.680011 13344 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "69e4a297899a4d6785d0745bbe5a3ffd" candidate_uuid: "e7a082460a324e97b17b1e7660ab3e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95463d19beac48dda9314dd2e97c2774" is_pre_election: true I0517 06:45:23.680424 13344 raft_consensus.cc:2180] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e7a082460a324e97b17b1e7660ab3e7b for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:45:23.680353 13594 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "69e4a297899a4d6785d0745bbe5a3ffd" candidate_uuid: "e7a082460a324e97b17b1e7660ab3e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b0d7495352c44fe87e7c441ff955e98" is_pre_election: true I0517 06:45:23.680835 13594 raft_consensus.cc:2149] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e7a082460a324e97b17b1e7660ab3e7b in current term 1: Already voted for candidate 4b0d7495352c44fe87e7c441ff955e98 in this term. I0517 06:45:23.681288 13399 leader_election.cc:393] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [CANDIDATE]: Term 1 pre-election: Vote denied by peer 95463d19beac48dda9314dd2e97c2774 (127.4.96.1:48936). Message: Invalid argument: T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e7a082460a324e97b17b1e7660ab3e7b for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:45:23.681705 13401 leader_election.cc:393] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [CANDIDATE]: Term 1 pre-election: Vote denied by peer 4b0d7495352c44fe87e7c441ff955e98 (127.4.96.3:59847). Message: Invalid argument: T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e7a082460a324e97b17b1e7660ab3e7b in current term 1: Already voted for candidate 4b0d7495352c44fe87e7c441ff955e98 in this term. I0517 06:45:23.681897 13401 leader_election.cc:270] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0517 06:45:23.682274 13687 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: Snoozing failure detection for 1.803s (election complete) I0517 06:45:23.682431 13687 raft_consensus.cc:2756] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:23.685211 13687 raft_consensus.cc:2459] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0517 06:45:24.001713 13692 raft_consensus.cc:435] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b0d7495352c44fe87e7c441ff955e98) I0517 06:45:24.001966 13692 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: Snoozing failure detection for 2.243s (starting election) I0517 06:45:24.002177 13692 raft_consensus.cc:457] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:24.003082 13692 leader_election.cc:243] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 4b0d7495352c44fe87e7c441ff955e98 (127.4.96.3:59847) I0517 06:45:24.003496 13692 leader_election.cc:243] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e7a082460a324e97b17b1e7660ab3e7b (127.4.96.2:56429) I0517 06:45:24.020925 13594 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "69e4a297899a4d6785d0745bbe5a3ffd" candidate_uuid: "95463d19beac48dda9314dd2e97c2774" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4b0d7495352c44fe87e7c441ff955e98" is_pre_election: true I0517 06:45:24.021361 13594 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Snoozing failure detection for 1.712s (vote granted) I0517 06:45:24.021580 13594 raft_consensus.cc:2219] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95463d19beac48dda9314dd2e97c2774 in term 1. I0517 06:45:24.021908 13468 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "69e4a297899a4d6785d0745bbe5a3ffd" candidate_uuid: "95463d19beac48dda9314dd2e97c2774" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e7a082460a324e97b17b1e7660ab3e7b" is_pre_election: true I0517 06:45:24.022322 13468 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: Snoozing failure detection for 2.828s (vote granted) I0517 06:45:24.022457 13276 leader_election.cc:379] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 4b0d7495352c44fe87e7c441ff955e98 (127.4.96.3:59847) I0517 06:45:24.022521 13468 raft_consensus.cc:2219] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95463d19beac48dda9314dd2e97c2774 in term 1. I0517 06:45:24.022675 13276 leader_election.cc:270] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0517 06:45:24.023126 13692 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: Snoozing failure detection for 1.763s (election complete) I0517 06:45:24.023314 13692 raft_consensus.cc:2511] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0517 06:45:24.023399 13278 leader_election.cc:379] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [CANDIDATE]: Term 2 pre-election: Vote granted by peer e7a082460a324e97b17b1e7660ab3e7b (127.4.96.2:56429) I0517 06:45:24.023510 13692 raft_consensus.cc:435] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4b0d7495352c44fe87e7c441ff955e98) I0517 06:45:24.023658 13692 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: Snoozing failure detection for 1.969s (starting election) I0517 06:45:24.023773 13692 raft_consensus.cc:2756] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:45:24.026581 13692 raft_consensus.cc:457] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:24.027178 13692 leader_election.cc:243] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [CANDIDATE]: Term 2 election: Requesting vote from peer 4b0d7495352c44fe87e7c441ff955e98 (127.4.96.3:59847) I0517 06:45:24.027444 13692 leader_election.cc:243] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [CANDIDATE]: Term 2 election: Requesting vote from peer e7a082460a324e97b17b1e7660ab3e7b (127.4.96.2:56429) I0517 06:45:24.028236 13594 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "69e4a297899a4d6785d0745bbe5a3ffd" candidate_uuid: "95463d19beac48dda9314dd2e97c2774" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4b0d7495352c44fe87e7c441ff955e98" I0517 06:45:24.028522 13594 raft_consensus.cc:2756] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:45:24.028460 13468 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "69e4a297899a4d6785d0745bbe5a3ffd" candidate_uuid: "95463d19beac48dda9314dd2e97c2774" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e7a082460a324e97b17b1e7660ab3e7b" I0517 06:45:24.028679 13594 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98: Snoozing failure detection for 1.679s (vote granted) I0517 06:45:24.028815 13468 raft_consensus.cc:2756] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [term 1 FOLLOWER]: Advancing to term 2 I0517 06:45:24.029006 13468 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b: Snoozing failure detection for 3.311s (vote granted) I0517 06:45:24.031342 13594 raft_consensus.cc:2219] T 69e4a297899a4d6785d0745bbe5a3ffd P 4b0d7495352c44fe87e7c441ff955e98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95463d19beac48dda9314dd2e97c2774 in term 2. I0517 06:45:24.031810 13468 raft_consensus.cc:2219] T 69e4a297899a4d6785d0745bbe5a3ffd P e7a082460a324e97b17b1e7660ab3e7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95463d19beac48dda9314dd2e97c2774 in term 2. I0517 06:45:24.032019 13276 leader_election.cc:379] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [CANDIDATE]: Term 2 election: Vote granted by peer 4b0d7495352c44fe87e7c441ff955e98 (127.4.96.3:59847) I0517 06:45:24.032198 13276 leader_election.cc:270] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0517 06:45:24.032459 13278 leader_election.cc:379] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [CANDIDATE]: Term 2 election: Vote granted by peer e7a082460a324e97b17b1e7660ab3e7b (127.4.96.2:56429) I0517 06:45:24.032559 13692 raft_consensus.cc:2704] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774: Snoozing failure detection for 1.510s (election complete) I0517 06:45:24.032773 13692 raft_consensus.cc:2511] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 2 FOLLOWER]: Leader election won for term 2 I0517 06:45:24.033812 13692 raft_consensus.cc:571] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [term 2 LEADER]: Becoming Leader. State: Replica: 95463d19beac48dda9314dd2e97c2774, State: Running, Role: LEADER I0517 06:45:24.034178 13692 consensus_queue.cc:204] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } } I0517 06:45:24.038203 13222 catalog_manager.cc:3686] T 69e4a297899a4d6785d0745bbe5a3ffd P 95463d19beac48dda9314dd2e97c2774 reported cstate change: term changed from 1 to 2, leader changed from 4b0d7495352c44fe87e7c441ff955e98 (127.4.96.3) to 95463d19beac48dda9314dd2e97c2774 (127.4.96.1). New cstate: current_term: 2 leader_uuid: "95463d19beac48dda9314dd2e97c2774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95463d19beac48dda9314dd2e97c2774" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48936 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b0d7495352c44fe87e7c441ff955e98" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 59847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7a082460a324e97b17b1e7660ab3e7b" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 56429 } health_report { overall_health: UNKNOWN } } } I0517 06:45:24.053622 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 13256 W0517 06:45:24.074414 13202 connection.cc:512] client connection to 127.4.96.1:48936 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:48936): Connection reset by peer (error 104) W0517 06:45:24.074537 13524 connection.cc:512] client connection to 127.4.96.1:48936 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:48936): Connection reset by peer (error 104) W0517 06:45:24.074611 13193 connection.cc:512] client connection to 127.4.96.1:48936 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:48936): Connection reset by peer (error 104) W0517 06:45:24.075354 13203 connection.cc:512] server connection from 127.4.96.1:51926 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:51926): Connection reset by peer (error 104) W0517 06:45:24.075625 13189 connection.cc:512] client connection to 127.4.96.1:48936 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:48936): Connection reset by peer (error 104) W0517 06:45:24.076427 13525 connection.cc:512] server connection from 127.4.96.1:34350 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:34350): Connection reset by peer (error 104) W0517 06:45:24.076480 13399 connection.cc:512] client connection to 127.4.96.1:48936 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:48936): Connection reset by peer (error 104) W0517 06:45:24.077301 13400 connection.cc:512] server connection from 127.4.96.1:54028 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:54028): Connection reset by peer (error 104) I0517 06:45:24.078092 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 13391 W0517 06:45:24.097306 13528 connection.cc:512] client connection to 127.4.96.2:56429 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:56429): Connection reset by peer (error 104) W0517 06:45:24.097322 13204 connection.cc:512] server connection from 127.4.96.2:39235 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:39235): Connection reset by peer (error 104) W0517 06:45:24.097425 13206 connection.cc:512] client connection to 127.4.96.2:56429 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:56429): Connection reset by peer (error 104) W0517 06:45:24.098186 13193 connection.cc:512] client connection to 127.4.96.2:56429 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:56429): Connection reset by peer (error 104) W0517 06:45:24.098923 13528 connection.cc:512] server connection from 127.4.96.2:41411 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:41411): Connection reset by peer (error 104) W0517 06:45:24.098985 13192 connection.cc:512] client connection to 127.4.96.2:56429 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:56429): Connection reset by peer (error 104) I0517 06:45:24.099830 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 13516 W0517 06:45:24.118132 13204 connection.cc:512] client connection to 127.4.96.3:59847 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:59847): Connection reset by peer (error 104) W0517 06:45:24.118484 13193 connection.cc:512] client connection to 127.4.96.3:59847 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:59847): Connection reset by peer (error 104) W0517 06:45:24.118129 13202 connection.cc:512] server connection from 127.4.96.3:34533 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:34533): Connection reset by peer (error 104) W0517 06:45:24.119385 13191 connection.cc:512] client connection to 127.4.96.3:59847 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:59847): Connection reset by peer (error 104) I0517 06:45:24.119473 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 13194 W0517 06:45:24.147292 13193 connection.cc:512] client connection to 127.4.96.62:34571 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:34571): Connection reset by peer (error 104) [ OK ] AdminCliTest.TestLeaderStepDown (4808 ms) [ RUN ] AdminCliTest.TestLeaderStepDownWhenNotPresent I0517 06:45:24.158592 4480 test_util.cc:200] Using random seed: 772977765 I0517 06:45:24.161610 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:45:24.161712 4480 ts_itest-base.cc:114] -------------- I0517 06:45:24.161815 4480 ts_itest-base.cc:115] 3 tablet servers I0517 06:45:24.161903 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:45:24.162005 4480 ts_itest-base.cc:117] -------------- I0517 06:45:24.164201 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:33601 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_wait_for_new_tablets_to_elect_leader=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:24.277269 13707 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:24.277587 13707 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:24.286522 13707 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:45:24.286644 13707 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:24.286739 13707 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:45:24.286849 13707 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:45:24.300750 13707 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --catalog_manager_wait_for_new_tablets_to_elect_leader=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:33601 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:24.308462 13707 master_main.cc:80] Initializing master server... I0517 06:45:24.309970 13707 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 161319us I0517 06:45:24.310741 13707 fs_manager.cc:260] Metadata directory not provided I0517 06:45:24.310863 13707 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:24.311051 13707 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:45:24.311153 13707 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:24.318404 13707 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "6830c6b57eac409ba6c5ea4fbefc8c64" format_stamp: "Formatted at 2018-05-17 06:45:24 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:24.319048 13707 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "6830c6b57eac409ba6c5ea4fbefc8c64" format_stamp: "Formatted at 2018-05-17 06:45:24 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:24.328579 13707 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:45:24.329702 13707 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:24.330103 13707 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:24.336449 13707 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:45:24.336557 13707 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "6830c6b57eac409ba6c5ea4fbefc8c64" format_stamp: "Formatted at 2018-05-17 06:45:24 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:24.336783 13707 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:24.509785 13707 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:24.512013 13707 master_main.cc:83] Starting Master server... I0517 06:45:24.555927 13707 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:33601 I0517 06:45:24.556766 13707 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:45:24.556876 13707 webserver.cc:184] Document root disabled I0517 06:45:24.558082 13707 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:41474/ I0517 06:45:24.558794 13707 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:45:24.561328 13769 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:24.569268 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 13707 I0517 06:45:24.570796 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:33601 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0517 06:45:24.577411 13769 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64: Bootstrap starting. I0517 06:45:24.579614 13769 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64: No blocks or log segments found. Creating new log. I0517 06:45:24.580749 13769 log.cc:520] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64: Log is configured to *not* fsync() on all Append() calls I0517 06:45:24.583097 13769 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64: No bootstrap required, opened a new log I0517 06:45:24.594921 13769 raft_consensus.cc:303] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6830c6b57eac409ba6c5ea4fbefc8c64" member_type: VOTER } I0517 06:45:24.595222 13769 raft_consensus.cc:329] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:24.595356 13769 raft_consensus.cc:604] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6830c6b57eac409ba6c5ea4fbefc8c64, State: Initialized, Role: FOLLOWER I0517 06:45:24.595788 13769 consensus_queue.cc:227] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6830c6b57eac409ba6c5ea4fbefc8c64" member_type: VOTER } I0517 06:45:24.596135 13769 raft_consensus.cc:343] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:45:24.596283 13769 raft_consensus.cc:435] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:45:24.596424 13769 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64: Snoozing failure detection for 1.647s (starting election) I0517 06:45:24.596563 13769 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:24.599050 13769 raft_consensus.cc:457] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6830c6b57eac409ba6c5ea4fbefc8c64" member_type: VOTER } I0517 06:45:24.599462 13769 leader_election.cc:270] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:24.601095 13772 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64: Snoozing failure detection for 2.049s (election complete) I0517 06:45:24.601428 13772 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:24.603070 13772 raft_consensus.cc:571] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [term 1 LEADER]: Becoming Leader. State: Replica: 6830c6b57eac409ba6c5ea4fbefc8c64, State: Running, Role: LEADER I0517 06:45:24.603510 13772 consensus_queue.cc:204] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6830c6b57eac409ba6c5ea4fbefc8c64" member_type: VOTER } I0517 06:45:24.606832 13773 sys_catalog.cc:335] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6830c6b57eac409ba6c5ea4fbefc8c64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6830c6b57eac409ba6c5ea4fbefc8c64" member_type: VOTER } } I0517 06:45:24.607573 13773 sys_catalog.cc:338] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [sys.catalog]: This master's current role is: LEADER WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:24.678033 13770 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0517 06:45:24.678387 13770 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:24.678589 13770 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:24.686632 13770 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:24.686966 13770 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:45:24.606889 13769 sys_catalog.cc:335] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "6830c6b57eac409ba6c5ea4fbefc8c64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6830c6b57eac409ba6c5ea4fbefc8c64" member_type: VOTER } } I0517 06:45:24.691870 13769 sys_catalog.cc:338] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [sys.catalog]: This master's current role is: LEADER I0517 06:45:24.693073 13769 sys_catalog.cc:422] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:45:24.607101 13774 sys_catalog.cc:335] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6830c6b57eac409ba6c5ea4fbefc8c64. Latest consensus state: current_term: 1 leader_uuid: "6830c6b57eac409ba6c5ea4fbefc8c64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6830c6b57eac409ba6c5ea4fbefc8c64" member_type: VOTER } } I0517 06:45:24.701201 13774 sys_catalog.cc:338] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64 [sys.catalog]: This master's current role is: LEADER I0517 06:45:24.703096 13707 master_main.cc:86] Master server successfully started. I0517 06:45:24.702976 13776 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:45:24.705065 13770 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:33601 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:24.708006 13776 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:45:24.717862 13770 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:24.720191 13770 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 161319us I0517 06:45:24.720783 13770 fs_manager.cc:260] Metadata directory not provided I0517 06:45:24.720970 13770 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:24.721240 13770 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:45:24.721423 13770 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:24.733700 13770 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "7af37c5629314932867fa97ffa7f0d6e" format_stamp: "Formatted at 2018-05-17 06:45:24 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:24.734637 13770 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "7af37c5629314932867fa97ffa7f0d6e" format_stamp: "Formatted at 2018-05-17 06:45:24 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:24.749106 13770 fs_manager.cc:495] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s I0517 06:45:24.750353 13770 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:24.750753 13770 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:24.755246 13776 catalog_manager.cc:899] Generated new certificate authority record I0517 06:45:24.757511 13776 catalog_manager.cc:1021] Loading token signing keys... I0517 06:45:24.759093 13770 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.008s sys 0.000s I0517 06:45:24.759264 13770 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "7af37c5629314932867fa97ffa7f0d6e" format_stamp: "Formatted at 2018-05-17 06:45:24 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:24.759606 13770 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:24.778921 13776 catalog_manager.cc:3962] T 00000000000000000000000000000000 P 6830c6b57eac409ba6c5ea4fbefc8c64: Generated new TSK 0 I0517 06:45:24.930028 13770 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:24.932600 13770 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:24.933748 13770 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:25.035406 13770 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:55444 I0517 06:45:25.036315 13770 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:45:25.036423 13770 webserver.cc:184] Document root disabled I0517 06:45:25.037833 13770 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:33479/ I0517 06:45:25.038549 13770 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:45:25.042618 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 13770 I0517 06:45:25.043803 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:33601 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0517 06:45:25.047284 13770 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:25.077754 13901 heartbeater.cc:346] Connected to a master server at 127.4.96.62:33601 I0517 06:45:25.078065 13901 heartbeater.cc:426] Registering TS with master... I0517 06:45:25.078651 13901 heartbeater.cc:475] Master 127.4.96.62:33601 requested a full tablet report, sending... I0517 06:45:25.080688 13736 ts_manager.cc:81] Registered new tserver with Master: 7af37c5629314932867fa97ffa7f0d6e (127.4.96.1:55444) I0517 06:45:25.083216 13736 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:52544 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:25.153843 13905 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0517 06:45:25.154208 13905 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:25.154417 13905 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:25.162325 13905 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:25.162680 13905 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:45:25.175609 13905 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:33601 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:25.183825 13905 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:25.185601 13905 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 161819us I0517 06:45:25.186095 13905 fs_manager.cc:260] Metadata directory not provided I0517 06:45:25.186218 13905 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:25.186419 13905 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:45:25.186539 13905 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:25.194074 13905 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "26b3eec87cbf40d5905f9222fdb43315" format_stamp: "Formatted at 2018-05-17 06:45:25 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:25.194733 13905 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "26b3eec87cbf40d5905f9222fdb43315" format_stamp: "Formatted at 2018-05-17 06:45:25 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:25.204470 13905 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:45:25.205569 13905 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:25.205996 13905 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:25.212512 13905 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:45:25.212623 13905 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "26b3eec87cbf40d5905f9222fdb43315" format_stamp: "Formatted at 2018-05-17 06:45:25 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:25.212848 13905 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:25.427326 13905 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:25.430229 13905 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:25.431345 13905 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:25.557893 13905 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:33031 I0517 06:45:25.558833 13905 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:45:25.558935 13905 webserver.cc:184] Document root disabled I0517 06:45:25.560307 13905 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:36473/ I0517 06:45:25.560995 13905 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:45:25.561698 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 13905 I0517 06:45:25.562976 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:33601 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0517 06:45:25.571013 13905 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:25.585155 14026 heartbeater.cc:346] Connected to a master server at 127.4.96.62:33601 I0517 06:45:25.585398 14026 heartbeater.cc:426] Registering TS with master... I0517 06:45:25.585853 14026 heartbeater.cc:475] Master 127.4.96.62:33601 requested a full tablet report, sending... I0517 06:45:25.587167 13736 ts_manager.cc:81] Registered new tserver with Master: 26b3eec87cbf40d5905f9222fdb43315 (127.4.96.2:33031) I0517 06:45:25.588237 13736 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:49400 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:25.718153 14029 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0517 06:45:25.718613 14029 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:25.718911 14029 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:25.733059 14029 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:25.733572 14029 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:45:25.754006 14029 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:33601 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:25.767334 14029 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:25.769760 14029 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 161819us I0517 06:45:25.770387 14029 fs_manager.cc:260] Metadata directory not provided I0517 06:45:25.770563 14029 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:25.770828 14029 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:45:25.770998 14029 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:25.783689 14029 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "7e1a132421e04eff80d291cd291adf7a" format_stamp: "Formatted at 2018-05-17 06:45:25 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:25.784651 14029 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "7e1a132421e04eff80d291cd291adf7a" format_stamp: "Formatted at 2018-05-17 06:45:25 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:25.798889 14029 fs_manager.cc:495] Time spent creating directory manager: real 0.014s user 0.016s sys 0.000s I0517 06:45:25.800609 14029 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:25.801143 14029 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:25.811594 14029 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:45:25.811740 14029 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "7e1a132421e04eff80d291cd291adf7a" format_stamp: "Formatted at 2018-05-17 06:45:25 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:25.812014 14029 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:25.991196 14029 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:25.994273 14029 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:25.995411 14029 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:26.086951 13901 heartbeater.cc:467] Master 127.4.96.62:33601 was elected leader, sending a full tablet report... I0517 06:45:26.118335 14029 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:39372 I0517 06:45:26.119323 14029 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:45:26.119427 14029 webserver.cc:184] Document root disabled I0517 06:45:26.120793 14029 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:52518/ I0517 06:45:26.121474 14029 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:45:26.122025 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 14029 I0517 06:45:26.143800 14029 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:26.163375 14150 heartbeater.cc:346] Connected to a master server at 127.4.96.62:33601 I0517 06:45:26.163664 14150 heartbeater.cc:426] Registering TS with master... I0517 06:45:26.164208 14150 heartbeater.cc:475] Master 127.4.96.62:33601 requested a full tablet report, sending... I0517 06:45:26.165678 13736 ts_manager.cc:81] Registered new tserver with Master: 7e1a132421e04eff80d291cd291adf7a (127.4.96.3:39372) I0517 06:45:26.167469 4480 external_mini_cluster.cc:482] 3 TS(s) registered with all masters I0517 06:45:26.167456 13736 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:60202 I0517 06:45:26.186664 13736 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55752: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0517 06:45:26.197336 13736 catalog_manager.cc:1463] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0517 06:45:26.261766 13963 tablet_service.cc:759] Processing CreateTablet for tablet 07cc6b3e80054ec99fbc1e7916e5deb5 (table=TestTable [id=d3cd8d748d7d4e41aa55bd7f50f6a504]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:26.262673 13963 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 07cc6b3e80054ec99fbc1e7916e5deb5. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:26.268127 13838 tablet_service.cc:759] Processing CreateTablet for tablet 07cc6b3e80054ec99fbc1e7916e5deb5 (table=TestTable [id=d3cd8d748d7d4e41aa55bd7f50f6a504]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:26.268718 14087 tablet_service.cc:759] Processing CreateTablet for tablet 07cc6b3e80054ec99fbc1e7916e5deb5 (table=TestTable [id=d3cd8d748d7d4e41aa55bd7f50f6a504]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:26.268990 13838 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 07cc6b3e80054ec99fbc1e7916e5deb5. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:26.269340 14087 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 07cc6b3e80054ec99fbc1e7916e5deb5. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:26.279161 13963 ts_tablet_manager.cc:1173] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 26b3eec87cbf40d5905f9222fdb43315: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:26.280630 14087 ts_tablet_manager.cc:1173] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7e1a132421e04eff80d291cd291adf7a: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:26.284929 13838 ts_tablet_manager.cc:1173] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7af37c5629314932867fa97ffa7f0d6e: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:26.286547 14171 ts_tablet_manager.cc:1028] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 26b3eec87cbf40d5905f9222fdb43315: Bootstrapping tablet I0517 06:45:26.287034 14171 tablet_bootstrap.cc:436] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 26b3eec87cbf40d5905f9222fdb43315: Bootstrap starting. I0517 06:45:26.288251 14173 ts_tablet_manager.cc:1028] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7af37c5629314932867fa97ffa7f0d6e: Bootstrapping tablet I0517 06:45:26.288765 14173 tablet_bootstrap.cc:436] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7af37c5629314932867fa97ffa7f0d6e: Bootstrap starting. I0517 06:45:26.289901 14171 tablet_bootstrap.cc:587] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 26b3eec87cbf40d5905f9222fdb43315: No blocks or log segments found. Creating new log. I0517 06:45:26.291371 14172 ts_tablet_manager.cc:1028] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7e1a132421e04eff80d291cd291adf7a: Bootstrapping tablet I0517 06:45:26.291600 14173 tablet_bootstrap.cc:587] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7af37c5629314932867fa97ffa7f0d6e: No blocks or log segments found. Creating new log. I0517 06:45:26.292665 14172 tablet_bootstrap.cc:436] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7e1a132421e04eff80d291cd291adf7a: Bootstrap starting. I0517 06:45:26.293153 14173 log.cc:520] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7af37c5629314932867fa97ffa7f0d6e: Log is configured to *not* fsync() on all Append() calls I0517 06:45:26.295523 14172 tablet_bootstrap.cc:587] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7e1a132421e04eff80d291cd291adf7a: No blocks or log segments found. Creating new log. I0517 06:45:26.295619 14173 tablet_bootstrap.cc:436] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7af37c5629314932867fa97ffa7f0d6e: No bootstrap required, opened a new log I0517 06:45:26.295954 14173 ts_tablet_manager.cc:1045] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7af37c5629314932867fa97ffa7f0d6e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s I0517 06:45:26.297132 14171 log.cc:520] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 26b3eec87cbf40d5905f9222fdb43315: Log is configured to *not* fsync() on all Append() calls I0517 06:45:26.299582 14172 log.cc:520] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7e1a132421e04eff80d291cd291adf7a: Log is configured to *not* fsync() on all Append() calls I0517 06:45:26.301725 14171 tablet_bootstrap.cc:436] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 26b3eec87cbf40d5905f9222fdb43315: No bootstrap required, opened a new log I0517 06:45:26.302108 14171 ts_tablet_manager.cc:1045] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 26b3eec87cbf40d5905f9222fdb43315: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s I0517 06:45:26.310777 14173 raft_consensus.cc:303] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7af37c5629314932867fa97ffa7f0d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1a132421e04eff80d291cd291adf7a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39372 } } peers { permanent_uuid: "26b3eec87cbf40d5905f9222fdb43315" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33031 } } peers { permanent_uuid: "7af37c5629314932867fa97ffa7f0d6e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55444 } } I0517 06:45:26.311512 14173 raft_consensus.cc:604] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7af37c5629314932867fa97ffa7f0d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af37c5629314932867fa97ffa7f0d6e, State: Initialized, Role: FOLLOWER I0517 06:45:26.311893 14173 consensus_queue.cc:227] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7af37c5629314932867fa97ffa7f0d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1a132421e04eff80d291cd291adf7a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39372 } } peers { permanent_uuid: "26b3eec87cbf40d5905f9222fdb43315" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33031 } } peers { permanent_uuid: "7af37c5629314932867fa97ffa7f0d6e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55444 } } I0517 06:45:26.312780 14172 tablet_bootstrap.cc:436] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7e1a132421e04eff80d291cd291adf7a: No bootstrap required, opened a new log I0517 06:45:26.313071 14172 ts_tablet_manager.cc:1045] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7e1a132421e04eff80d291cd291adf7a: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s I0517 06:45:26.317857 14173 ts_tablet_manager.cc:1073] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7af37c5629314932867fa97ffa7f0d6e: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s I0517 06:45:26.317596 14171 raft_consensus.cc:303] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 26b3eec87cbf40d5905f9222fdb43315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1a132421e04eff80d291cd291adf7a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39372 } } peers { permanent_uuid: "26b3eec87cbf40d5905f9222fdb43315" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33031 } } peers { permanent_uuid: "7af37c5629314932867fa97ffa7f0d6e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55444 } } I0517 06:45:26.318317 14171 raft_consensus.cc:604] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 26b3eec87cbf40d5905f9222fdb43315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b3eec87cbf40d5905f9222fdb43315, State: Initialized, Role: FOLLOWER I0517 06:45:26.318615 14171 consensus_queue.cc:227] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 26b3eec87cbf40d5905f9222fdb43315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1a132421e04eff80d291cd291adf7a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39372 } } peers { permanent_uuid: "26b3eec87cbf40d5905f9222fdb43315" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33031 } } peers { permanent_uuid: "7af37c5629314932867fa97ffa7f0d6e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55444 } } I0517 06:45:26.324484 14026 heartbeater.cc:467] Master 127.4.96.62:33601 was elected leader, sending a full tablet report... I0517 06:45:26.325553 14172 raft_consensus.cc:303] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7e1a132421e04eff80d291cd291adf7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1a132421e04eff80d291cd291adf7a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39372 } } peers { permanent_uuid: "26b3eec87cbf40d5905f9222fdb43315" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33031 } } peers { permanent_uuid: "7af37c5629314932867fa97ffa7f0d6e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55444 } } I0517 06:45:26.326102 14171 ts_tablet_manager.cc:1073] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 26b3eec87cbf40d5905f9222fdb43315: Time spent starting tablet: real 0.024s user 0.012s sys 0.012s I0517 06:45:26.326349 14172 raft_consensus.cc:604] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7e1a132421e04eff80d291cd291adf7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e1a132421e04eff80d291cd291adf7a, State: Initialized, Role: FOLLOWER I0517 06:45:26.326722 14172 consensus_queue.cc:227] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7e1a132421e04eff80d291cd291adf7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1a132421e04eff80d291cd291adf7a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 39372 } } peers { permanent_uuid: "26b3eec87cbf40d5905f9222fdb43315" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 33031 } } peers { permanent_uuid: "7af37c5629314932867fa97ffa7f0d6e" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55444 } } I0517 06:45:26.328795 14150 heartbeater.cc:467] Master 127.4.96.62:33601 was elected leader, sending a full tablet report... I0517 06:45:26.329461 4480 external_mini_cluster.cc:482] 3 TS(s) registered with all masters I0517 06:45:26.330430 14172 ts_tablet_manager.cc:1073] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7e1a132421e04eff80d291cd291adf7a: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s I0517 06:45:26.332062 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7af37c5629314932867fa97ffa7f0d6e to finish bootstrapping I0517 06:45:26.341337 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 26b3eec87cbf40d5905f9222fdb43315 to finish bootstrapping I0517 06:45:26.349617 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7e1a132421e04eff80d291cd291adf7a to finish bootstrapping W0517 06:45:26.393851 14151 tablet.cc:1678] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7e1a132421e04eff80d291cd291adf7a: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:45:26.502271 14181 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:26.548584 13902 tablet.cc:1678] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 7af37c5629314932867fa97ffa7f0d6e: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:45:26.572592 14027 tablet.cc:1678] T 07cc6b3e80054ec99fbc1e7916e5deb5 P 26b3eec87cbf40d5905f9222fdb43315: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:27.708694 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 13770 W0517 06:45:27.729188 13719 connection.cc:512] client connection to 127.4.96.1:55444 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:55444): Connection reset by peer (error 104) W0517 06:45:27.729396 13706 connection.cc:512] client connection to 127.4.96.1:55444 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:55444): Connection reset by peer (error 104) W0517 06:45:27.730207 13717 connection.cc:512] server connection from 127.4.96.1:52544 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:52544): Connection reset by peer (error 104) W0517 06:45:27.730515 13705 connection.cc:512] client connection to 127.4.96.1:55444 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:55444): Connection reset by peer (error 104) I0517 06:45:27.731474 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 13905 W0517 06:45:27.749418 13719 connection.cc:512] server connection from 127.4.96.2:49400 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:49400): Connection reset by peer (error 104) I0517 06:45:27.749550 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 14029 W0517 06:45:27.749604 13702 connection.cc:512] client connection to 127.4.96.2:33031 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:33031): Connection reset by peer (error 104) W0517 06:45:27.749955 13715 connection.cc:512] client connection to 127.4.96.2:33031 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:33031): Connection reset by peer (error 104) W0517 06:45:27.750473 13706 connection.cc:512] client connection to 127.4.96.2:33031 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:33031): Connection reset by peer (error 104) W0517 06:45:27.767223 13717 connection.cc:512] client connection to 127.4.96.3:39372 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:39372): Connection reset by peer (error 104) W0517 06:45:27.767336 13704 connection.cc:512] client connection to 127.4.96.3:39372 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:39372): Connection reset by peer (error 104) I0517 06:45:27.767391 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 13707 W0517 06:45:27.768179 13706 connection.cc:512] client connection to 127.4.96.3:39372 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:39372): Connection reset by peer (error 104) W0517 06:45:27.797448 13706 connection.cc:512] client connection to 127.4.96.62:33601 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:33601): Connection reset by peer (error 104) [ OK ] AdminCliTest.TestLeaderStepDownWhenNotPresent (3652 ms) [ RUN ] AdminCliTest.TestDeleteTable I0517 06:45:27.809957 4480 test_util.cc:200] Using random seed: 776629111 I0517 06:45:27.812870 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:45:27.812970 4480 ts_itest-base.cc:114] -------------- I0517 06:45:27.813056 4480 ts_itest-base.cc:115] 1 tablet servers I0517 06:45:27.813141 4480 ts_itest-base.cc:116] 1 replicas per TS I0517 06:45:27.813225 4480 ts_itest-base.cc:117] -------------- I0517 06:45:27.815533 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:37260 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:27.928311 14195 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:27.928633 14195 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:27.937558 14195 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:45:27.937686 14195 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:27.937801 14195 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:45:27.937943 14195 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:45:27.951468 14195 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:37260 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:27.959100 14195 master_main.cc:80] Initializing master server... I0517 06:45:27.960604 14195 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 162819us I0517 06:45:27.961341 14195 fs_manager.cc:260] Metadata directory not provided I0517 06:45:27.961472 14195 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:27.961678 14195 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:45:27.961791 14195 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:27.969038 14195 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "bcade0038ff04215b52157779c98571a" format_stamp: "Formatted at 2018-05-17 06:45:27 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:27.969708 14195 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "bcade0038ff04215b52157779c98571a" format_stamp: "Formatted at 2018-05-17 06:45:27 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:27.979507 14195 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:45:27.980621 14195 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:27.981027 14195 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:27.987385 14195 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:45:27.987498 14195 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "bcade0038ff04215b52157779c98571a" format_stamp: "Formatted at 2018-05-17 06:45:27 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:27.987694 14195 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:28.143658 14195 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:28.145313 14195 master_main.cc:83] Starting Master server... I0517 06:45:28.185473 14195 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:37260 I0517 06:45:28.186337 14195 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:45:28.186444 14195 webserver.cc:184] Document root disabled I0517 06:45:28.187652 14195 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:45866/ I0517 06:45:28.188354 14195 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:45:28.190892 14256 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:28.197913 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 14195 I0517 06:45:28.198917 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:37260 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:28.204401 14256 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a: Bootstrap starting. I0517 06:45:28.206571 14256 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a: No blocks or log segments found. Creating new log. I0517 06:45:28.207717 14256 log.cc:520] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a: Log is configured to *not* fsync() on all Append() calls I0517 06:45:28.210058 14256 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a: No bootstrap required, opened a new log I0517 06:45:28.221731 14256 raft_consensus.cc:303] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcade0038ff04215b52157779c98571a" member_type: VOTER } I0517 06:45:28.222041 14256 raft_consensus.cc:329] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:28.222218 14256 raft_consensus.cc:604] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcade0038ff04215b52157779c98571a, State: Initialized, Role: FOLLOWER I0517 06:45:28.222743 14256 consensus_queue.cc:227] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcade0038ff04215b52157779c98571a" member_type: VOTER } I0517 06:45:28.223184 14256 raft_consensus.cc:343] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:45:28.223392 14256 raft_consensus.cc:435] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:45:28.223587 14256 raft_consensus.cc:2704] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a: Snoozing failure detection for 1.747s (starting election) I0517 06:45:28.223755 14256 raft_consensus.cc:2756] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:28.226148 14256 raft_consensus.cc:457] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcade0038ff04215b52157779c98571a" member_type: VOTER } I0517 06:45:28.226564 14256 leader_election.cc:270] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:28.228096 14259 raft_consensus.cc:2704] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a: Snoozing failure detection for 2.124s (election complete) I0517 06:45:28.229360 14259 raft_consensus.cc:2511] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:28.246403 14259 raft_consensus.cc:571] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [term 1 LEADER]: Becoming Leader. State: Replica: bcade0038ff04215b52157779c98571a, State: Running, Role: LEADER I0517 06:45:28.246800 14259 consensus_queue.cc:204] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcade0038ff04215b52157779c98571a" member_type: VOTER } I0517 06:45:28.250162 14256 sys_catalog.cc:335] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "bcade0038ff04215b52157779c98571a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcade0038ff04215b52157779c98571a" member_type: VOTER } } I0517 06:45:28.250577 14256 sys_catalog.cc:338] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [sys.catalog]: This master's current role is: LEADER I0517 06:45:28.250656 14260 sys_catalog.cc:335] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcade0038ff04215b52157779c98571a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcade0038ff04215b52157779c98571a" member_type: VOTER } } I0517 06:45:28.310395 14260 sys_catalog.cc:338] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [sys.catalog]: This master's current role is: LEADER I0517 06:45:28.315449 14256 sys_catalog.cc:422] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [sys.catalog]: configured and running, proceeding with master startup. I0517 06:45:28.251835 14261 sys_catalog.cc:335] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcade0038ff04215b52157779c98571a. Latest consensus state: current_term: 1 leader_uuid: "bcade0038ff04215b52157779c98571a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcade0038ff04215b52157779c98571a" member_type: VOTER } } I0517 06:45:28.317744 14261 sys_catalog.cc:338] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a [sys.catalog]: This master's current role is: LEADER I0517 06:45:28.318876 14195 master_main.cc:86] Master server successfully started. W0517 06:45:28.322177 14264 catalog_manager.cc:1047] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0517 06:45:28.322418 14264 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0517 06:45:28.322688 14263 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:45:28.325634 14263 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:28.341519 14257 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:28.341861 14257 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:28.349764 14257 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:28.350111 14257 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:45:28.367933 14263 catalog_manager.cc:899] Generated new certificate authority record I0517 06:45:28.368947 14257 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:37260 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:28.369720 14263 catalog_manager.cc:1021] Loading token signing keys... I0517 06:45:28.377071 14257 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:28.378893 14257 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 163319us I0517 06:45:28.379313 14257 fs_manager.cc:260] Metadata directory not provided I0517 06:45:28.379444 14257 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:28.379655 14257 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:45:28.379777 14257 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:28.388065 14263 catalog_manager.cc:3962] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a: Generated new TSK 0 I0517 06:45:28.388147 14257 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "fa607ff603494b5788bfc8dbf7f8f14d" format_stamp: "Formatted at 2018-05-17 06:45:28 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:28.388844 14257 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "fa607ff603494b5788bfc8dbf7f8f14d" format_stamp: "Formatted at 2018-05-17 06:45:28 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:28.399634 14257 fs_manager.cc:495] Time spent creating directory manager: real 0.010s user 0.004s sys 0.008s I0517 06:45:28.400813 14257 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:28.401232 14257 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:28.407717 14257 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:45:28.407829 14257 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "fa607ff603494b5788bfc8dbf7f8f14d" format_stamp: "Formatted at 2018-05-17 06:45:28 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:28.408030 14257 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:28.630237 14257 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:28.632855 14257 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:28.634032 14257 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:28.735477 14257 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:40122 I0517 06:45:28.736440 14257 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:45:28.736548 14257 webserver.cc:184] Document root disabled I0517 06:45:28.737900 14257 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:39786/ I0517 06:45:28.738615 14257 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:45:28.745129 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 14257 I0517 06:45:28.754815 14257 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:28.772214 14388 heartbeater.cc:346] Connected to a master server at 127.4.96.62:37260 I0517 06:45:28.772505 14388 heartbeater.cc:426] Registering TS with master... I0517 06:45:28.773072 14388 heartbeater.cc:475] Master 127.4.96.62:37260 requested a full tablet report, sending... I0517 06:45:28.775142 14222 ts_manager.cc:81] Registered new tserver with Master: fa607ff603494b5788bfc8dbf7f8f14d (127.4.96.1:40122) I0517 06:45:28.777253 14222 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:43043 I0517 06:45:28.777698 4480 external_mini_cluster.cc:482] 1 TS(s) registered with all masters I0517 06:45:28.795524 14222 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34715: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:45:28.826203 14325 tablet_service.cc:759] Processing CreateTablet for tablet 80b49b735f0a40bf98bd9fdb63f5442f (table=TestTable [id=17cf1799958646f5a065de9797183f85]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:28.826867 14325 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 80b49b735f0a40bf98bd9fdb63f5442f. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:28.838768 14325 ts_tablet_manager.cc:1173] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:28.840337 14401 ts_tablet_manager.cc:1028] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: Bootstrapping tablet I0517 06:45:28.840790 14401 tablet_bootstrap.cc:436] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: Bootstrap starting. I0517 06:45:28.843956 14401 tablet_bootstrap.cc:587] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: No blocks or log segments found. Creating new log. I0517 06:45:28.845571 14401 log.cc:520] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: Log is configured to *not* fsync() on all Append() calls I0517 06:45:28.848049 14401 tablet_bootstrap.cc:436] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: No bootstrap required, opened a new log I0517 06:45:28.848420 14401 ts_tablet_manager.cc:1045] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s I0517 06:45:28.863955 14401 raft_consensus.cc:303] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa607ff603494b5788bfc8dbf7f8f14d" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 40122 } } I0517 06:45:28.864516 14401 raft_consensus.cc:329] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:28.864708 14401 raft_consensus.cc:604] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa607ff603494b5788bfc8dbf7f8f14d, State: Initialized, Role: FOLLOWER I0517 06:45:28.865214 14401 consensus_queue.cc:227] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa607ff603494b5788bfc8dbf7f8f14d" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 40122 } } I0517 06:45:28.865710 14401 raft_consensus.cc:343] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:45:28.865913 14401 raft_consensus.cc:435] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:45:28.866139 14401 raft_consensus.cc:2704] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: Snoozing failure detection for 1.963s (starting election) I0517 06:45:28.866358 14401 raft_consensus.cc:2756] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:28.870244 14401 raft_consensus.cc:457] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa607ff603494b5788bfc8dbf7f8f14d" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 40122 } } I0517 06:45:28.870826 14401 leader_election.cc:270] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:28.872448 14403 raft_consensus.cc:2704] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: Snoozing failure detection for 2.018s (election complete) I0517 06:45:28.872645 14403 raft_consensus.cc:2511] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:28.876523 14403 raft_consensus.cc:571] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [term 1 LEADER]: Becoming Leader. State: Replica: fa607ff603494b5788bfc8dbf7f8f14d, State: Running, Role: LEADER I0517 06:45:28.876910 14403 consensus_queue.cc:204] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa607ff603494b5788bfc8dbf7f8f14d" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 40122 } } I0517 06:45:28.876833 14388 heartbeater.cc:467] Master 127.4.96.62:37260 was elected leader, sending a full tablet report... I0517 06:45:28.955862 14222 catalog_manager.cc:3686] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d reported cstate change: term changed from 0 to 1, leader changed from to fa607ff603494b5788bfc8dbf7f8f14d (127.4.96.1). New cstate: current_term: 1 leader_uuid: "fa607ff603494b5788bfc8dbf7f8f14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa607ff603494b5788bfc8dbf7f8f14d" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 40122 } health_report { overall_health: HEALTHY } } } I0517 06:45:28.958079 14401 ts_tablet_manager.cc:1073] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: Time spent starting tablet: real 0.109s user 0.024s sys 0.008s I0517 06:45:28.980473 4480 external_mini_cluster.cc:482] 1 TS(s) registered with all masters I0517 06:45:28.982442 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fa607ff603494b5788bfc8dbf7f8f14d to finish bootstrapping W0517 06:45:28.996469 14389 tablet.cc:1678] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:45:29.176224 14412 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:45:29.446683 14222 catalog_manager.cc:1706] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34719: table { table_name: "TestTable" } I0517 06:45:29.450929 14222 catalog_manager.cc:3897] T 00000000000000000000000000000000 P bcade0038ff04215b52157779c98571a: Sending DeleteTablet for 1 replicas of tablet 80b49b735f0a40bf98bd9fdb63f5442f I0517 06:45:29.451184 14222 catalog_manager.cc:3013] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 80b49b735f0a40bf98bd9fdb63f5442f on fa607ff603494b5788bfc8dbf7f8f14d (127.4.96.1:40122) (Table deleted at 2018-05-17 06:45:29 UTC) I0517 06:45:29.452410 14325 tablet_service.cc:800] Processing DeleteTablet for tablet 80b49b735f0a40bf98bd9fdb63f5442f with delete_type TABLET_DATA_DELETED (Table deleted at 2018-05-17 06:45:29 UTC) from {username='slave'} at 127.0.0.1:49826 I0517 06:45:29.454058 14421 tablet_replica.cc:261] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: stopping tablet replica I0517 06:45:29.454377 14421 maintenance_manager.cc:237] P fa607ff603494b5788bfc8dbf7f8f14d: Unregistered op CompactRowSetsOp(80b49b735f0a40bf98bd9fdb63f5442f) I0517 06:45:29.454603 14421 maintenance_manager.cc:237] P fa607ff603494b5788bfc8dbf7f8f14d: Unregistered op MinorDeltaCompactionOp(80b49b735f0a40bf98bd9fdb63f5442f) I0517 06:45:29.454795 14421 maintenance_manager.cc:237] P fa607ff603494b5788bfc8dbf7f8f14d: Unregistered op MajorDeltaCompactionOp(80b49b735f0a40bf98bd9fdb63f5442f) I0517 06:45:29.454977 14421 maintenance_manager.cc:237] P fa607ff603494b5788bfc8dbf7f8f14d: Unregistered op UndoDeltaBlockGCOp(80b49b735f0a40bf98bd9fdb63f5442f) I0517 06:45:29.455214 14421 maintenance_manager.cc:237] P fa607ff603494b5788bfc8dbf7f8f14d: Unregistered op FlushMRSOp(80b49b735f0a40bf98bd9fdb63f5442f) I0517 06:45:29.455411 14421 maintenance_manager.cc:237] P fa607ff603494b5788bfc8dbf7f8f14d: Unregistered op FlushDeltaMemStoresOp(80b49b735f0a40bf98bd9fdb63f5442f) I0517 06:45:29.455588 14421 maintenance_manager.cc:237] P fa607ff603494b5788bfc8dbf7f8f14d: Unregistered op LogGCOp(80b49b735f0a40bf98bd9fdb63f5442f) I0517 06:45:29.455814 14421 raft_consensus.cc:2016] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [term 1 LEADER]: Raft consensus shutting down. I0517 06:45:29.456146 14421 raft_consensus.cc:2043] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d [term 1 FOLLOWER]: Raft consensus is shut down! I0517 06:45:29.459105 14421 ts_tablet_manager.cc:1369] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: Deleting tablet data with delete state TABLET_DATA_DELETED I0517 06:45:29.467787 14421 ts_tablet_manager.cc:1382] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 I0517 06:45:29.468021 14421 log.cc:981] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: Deleting WAL directory at /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/wals/80b49b735f0a40bf98bd9fdb63f5442f I0517 06:45:29.468583 14421 ts_tablet_manager.cc:1402] T 80b49b735f0a40bf98bd9fdb63f5442f P fa607ff603494b5788bfc8dbf7f8f14d: Deleting consensus metadata I0517 06:45:29.470788 14204 catalog_manager.cc:2991] TS fa607ff603494b5788bfc8dbf7f8f14d (127.4.96.1:40122): tablet 80b49b735f0a40bf98bd9fdb63f5442f (table TestTable [id=17cf1799958646f5a065de9797183f85]) successfully deleted I0517 06:45:30.467231 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 14257 W0517 06:45:30.487107 14204 connection.cc:512] client connection to 127.4.96.1:40122 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:40122): Connection reset by peer (error 104) W0517 06:45:30.487159 14203 connection.cc:512] server connection from 127.4.96.1:43043 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:43043): Connection reset by peer (error 104) W0517 06:45:30.487861 14194 connection.cc:512] client connection to 127.4.96.1:40122 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:40122): Connection reset by peer (error 104) I0517 06:45:30.488979 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 14195 W0517 06:45:30.511006 14194 connection.cc:512] client connection to 127.4.96.62:37260 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:37260): Connection reset by peer (error 104) [ OK ] AdminCliTest.TestDeleteTable (2710 ms) [ RUN ] AdminCliTest.TestListTables I0517 06:45:30.521353 4480 test_util.cc:200] Using random seed: 779340527 I0517 06:45:30.525053 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:45:30.525184 4480 ts_itest-base.cc:114] -------------- I0517 06:45:30.525281 4480 ts_itest-base.cc:115] 1 tablet servers I0517 06:45:30.525377 4480 ts_itest-base.cc:116] 1 replicas per TS I0517 06:45:30.525475 4480 ts_itest-base.cc:117] -------------- I0517 06:45:30.528261 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:56237 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:30.640475 14427 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:30.640785 14427 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:30.649657 14427 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:45:30.649775 14427 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:30.649870 14427 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:45:30.649991 14427 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:45:30.663413 14427 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:56237 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:30.670984 14427 master_main.cc:80] Initializing master server... I0517 06:45:30.672456 14427 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 164319us I0517 06:45:30.673177 14427 fs_manager.cc:260] Metadata directory not provided I0517 06:45:30.673297 14427 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:30.673483 14427 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:45:30.673586 14427 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:30.680671 14427 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "2e7c090a194d4cc38e49465ffa8fbea1" format_stamp: "Formatted at 2018-05-17 06:45:30 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:30.681295 14427 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "2e7c090a194d4cc38e49465ffa8fbea1" format_stamp: "Formatted at 2018-05-17 06:45:30 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:30.690682 14427 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:45:30.691787 14427 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:30.692184 14427 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:30.698499 14427 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:45:30.698611 14427 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "2e7c090a194d4cc38e49465ffa8fbea1" format_stamp: "Formatted at 2018-05-17 06:45:30 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:30.698819 14427 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:30.904770 14427 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:30.906440 14427 master_main.cc:83] Starting Master server... I0517 06:45:30.945600 14427 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:56237 I0517 06:45:30.946470 14427 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:45:30.946575 14427 webserver.cc:184] Document root disabled I0517 06:45:30.947790 14427 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:40755/ I0517 06:45:30.948488 14427 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:45:30.950618 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 14427 I0517 06:45:30.951059 14488 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:30.951593 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:56237 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:30.964591 14488 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1: Bootstrap starting. I0517 06:45:30.966771 14488 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1: No blocks or log segments found. Creating new log. I0517 06:45:30.967895 14488 log.cc:520] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1: Log is configured to *not* fsync() on all Append() calls I0517 06:45:30.970320 14488 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1: No bootstrap required, opened a new log I0517 06:45:30.981885 14488 raft_consensus.cc:303] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7c090a194d4cc38e49465ffa8fbea1" member_type: VOTER } I0517 06:45:30.982208 14488 raft_consensus.cc:329] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:30.982347 14488 raft_consensus.cc:604] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e7c090a194d4cc38e49465ffa8fbea1, State: Initialized, Role: FOLLOWER I0517 06:45:30.982801 14488 consensus_queue.cc:227] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7c090a194d4cc38e49465ffa8fbea1" member_type: VOTER } I0517 06:45:30.983158 14488 raft_consensus.cc:343] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:45:30.983311 14488 raft_consensus.cc:435] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:45:30.983466 14488 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1: Snoozing failure detection for 2.099s (starting election) I0517 06:45:30.983620 14488 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:30.986064 14488 raft_consensus.cc:457] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7c090a194d4cc38e49465ffa8fbea1" member_type: VOTER } I0517 06:45:30.986488 14488 leader_election.cc:270] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:30.988093 14491 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1: Snoozing failure detection for 1.589s (election complete) I0517 06:45:30.988451 14491 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:30.990046 14491 raft_consensus.cc:571] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [term 1 LEADER]: Becoming Leader. State: Replica: 2e7c090a194d4cc38e49465ffa8fbea1, State: Running, Role: LEADER I0517 06:45:30.990368 14491 consensus_queue.cc:204] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7c090a194d4cc38e49465ffa8fbea1" member_type: VOTER } I0517 06:45:30.993430 14492 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e7c090a194d4cc38e49465ffa8fbea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7c090a194d4cc38e49465ffa8fbea1" member_type: VOTER } } I0517 06:45:30.993877 14492 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [sys.catalog]: This master's current role is: LEADER WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:31.057276 14489 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:31.057616 14489 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:31.065575 14489 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:31.065908 14489 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:45:30.994242 14493 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e7c090a194d4cc38e49465ffa8fbea1. Latest consensus state: current_term: 1 leader_uuid: "2e7c090a194d4cc38e49465ffa8fbea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7c090a194d4cc38e49465ffa8fbea1" member_type: VOTER } } I0517 06:45:31.078120 14493 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [sys.catalog]: This master's current role is: LEADER I0517 06:45:31.079452 14489 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:56237 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:30.995211 14488 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "2e7c090a194d4cc38e49465ffa8fbea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7c090a194d4cc38e49465ffa8fbea1" member_type: VOTER } } I0517 06:45:31.088146 14488 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [sys.catalog]: This master's current role is: LEADER I0517 06:45:31.089354 14488 sys_catalog.cc:422] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:45:31.089540 14489 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:31.089552 14495 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:45:31.091696 14427 master_main.cc:86] Master server successfully started. I0517 06:45:31.091903 14489 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 164819us I0517 06:45:31.092403 14489 fs_manager.cc:260] Metadata directory not provided I0517 06:45:31.092602 14489 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:31.092893 14489 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:45:31.093068 14489 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:31.094887 14495 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:45:31.105608 14489 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "d52ee272c73d4ceea9ea25bee22cf745" format_stamp: "Formatted at 2018-05-17 06:45:31 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:31.106531 14489 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "d52ee272c73d4ceea9ea25bee22cf745" format_stamp: "Formatted at 2018-05-17 06:45:31 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:31.136952 14489 fs_manager.cc:495] Time spent creating directory manager: real 0.030s user 0.016s sys 0.012s I0517 06:45:31.138649 14489 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:31.139176 14489 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:31.151386 14489 fs_manager.cc:417] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s I0517 06:45:31.151554 14489 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "d52ee272c73d4ceea9ea25bee22cf745" format_stamp: "Formatted at 2018-05-17 06:45:31 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:31.151831 14489 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:31.152197 14495 catalog_manager.cc:899] Generated new certificate authority record I0517 06:45:31.153637 14495 catalog_manager.cc:1021] Loading token signing keys... I0517 06:45:31.161562 14495 catalog_manager.cc:3962] T 00000000000000000000000000000000 P 2e7c090a194d4cc38e49465ffa8fbea1: Generated new TSK 0 I0517 06:45:31.353229 14489 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:31.356401 14489 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:31.357517 14489 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:31.511273 14489 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:48897 I0517 06:45:31.512609 14489 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:45:31.512760 14489 webserver.cc:184] Document root disabled I0517 06:45:31.514237 14489 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:37234/ I0517 06:45:31.515090 14489 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTables.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:45:31.516782 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 14489 I0517 06:45:31.526924 14489 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:31.553161 14620 heartbeater.cc:346] Connected to a master server at 127.4.96.62:56237 I0517 06:45:31.553453 14620 heartbeater.cc:426] Registering TS with master... I0517 06:45:31.554054 14620 heartbeater.cc:475] Master 127.4.96.62:56237 requested a full tablet report, sending... I0517 06:45:31.555850 14455 ts_manager.cc:81] Registered new tserver with Master: d52ee272c73d4ceea9ea25bee22cf745 (127.4.96.1:48897) I0517 06:45:31.557726 14455 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:34039 I0517 06:45:31.559517 4480 external_mini_cluster.cc:482] 1 TS(s) registered with all masters I0517 06:45:31.577183 14455 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34658: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:45:31.608903 14557 tablet_service.cc:759] Processing CreateTablet for tablet 7e9595a18b8040b2890a89e12836f8d1 (table=TestTable [id=8427b17012e346bc92cd5cc1d77d4818]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:31.609565 14557 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7e9595a18b8040b2890a89e12836f8d1. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:31.621350 14557 ts_tablet_manager.cc:1173] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:31.623668 14633 ts_tablet_manager.cc:1028] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745: Bootstrapping tablet I0517 06:45:31.624132 14633 tablet_bootstrap.cc:436] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745: Bootstrap starting. I0517 06:45:31.628185 14633 tablet_bootstrap.cc:587] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745: No blocks or log segments found. Creating new log. I0517 06:45:31.629760 14633 log.cc:520] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745: Log is configured to *not* fsync() on all Append() calls I0517 06:45:31.632376 14633 tablet_bootstrap.cc:436] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745: No bootstrap required, opened a new log I0517 06:45:31.632725 14633 ts_tablet_manager.cc:1045] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s I0517 06:45:31.648262 14633 raft_consensus.cc:303] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52ee272c73d4ceea9ea25bee22cf745" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48897 } } I0517 06:45:31.648810 14633 raft_consensus.cc:329] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:31.649001 14633 raft_consensus.cc:604] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d52ee272c73d4ceea9ea25bee22cf745, State: Initialized, Role: FOLLOWER I0517 06:45:31.649628 14633 consensus_queue.cc:227] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52ee272c73d4ceea9ea25bee22cf745" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48897 } } I0517 06:45:31.650177 14633 raft_consensus.cc:343] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:45:31.650413 14633 raft_consensus.cc:435] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:45:31.650627 14633 raft_consensus.cc:2704] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745: Snoozing failure detection for 1.585s (starting election) I0517 06:45:31.650831 14633 raft_consensus.cc:2756] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:31.654844 14633 raft_consensus.cc:457] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52ee272c73d4ceea9ea25bee22cf745" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48897 } } I0517 06:45:31.655442 14633 leader_election.cc:270] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:31.656940 14635 raft_consensus.cc:2704] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745: Snoozing failure detection for 1.505s (election complete) I0517 06:45:31.657119 14635 raft_consensus.cc:2511] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:31.659112 14635 raft_consensus.cc:571] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745 [term 1 LEADER]: Becoming Leader. State: Replica: d52ee272c73d4ceea9ea25bee22cf745, State: Running, Role: LEADER I0517 06:45:31.659554 14635 consensus_queue.cc:204] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52ee272c73d4ceea9ea25bee22cf745" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48897 } } I0517 06:45:31.659456 14620 heartbeater.cc:467] Master 127.4.96.62:56237 was elected leader, sending a full tablet report... I0517 06:45:31.733525 14633 ts_tablet_manager.cc:1073] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745: Time spent starting tablet: real 0.101s user 0.036s sys 0.064s I0517 06:45:31.736047 14455 catalog_manager.cc:3686] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745 reported cstate change: term changed from 0 to 1, leader changed from to d52ee272c73d4ceea9ea25bee22cf745 (127.4.96.1). New cstate: current_term: 1 leader_uuid: "d52ee272c73d4ceea9ea25bee22cf745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52ee272c73d4ceea9ea25bee22cf745" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 48897 } health_report { overall_health: HEALTHY } } } I0517 06:45:31.757980 4480 external_mini_cluster.cc:482] 1 TS(s) registered with all masters I0517 06:45:31.760030 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d52ee272c73d4ceea9ea25bee22cf745 to finish bootstrapping W0517 06:45:31.776932 14621 tablet.cc:1678] T 7e9595a18b8040b2890a89e12836f8d1 P d52ee272c73d4ceea9ea25bee22cf745: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:45:31.883416 14639 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:45:33.149271 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 14489 W0517 06:45:33.169327 14437 connection.cc:512] client connection to 127.4.96.1:48897 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:48897): Connection reset by peer (error 104) W0517 06:45:33.169347 14439 connection.cc:512] server connection from 127.4.96.1:34039 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:34039): Connection reset by peer (error 104) W0517 06:45:33.169374 14426 connection.cc:512] client connection to 127.4.96.1:48897 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:48897): Connection reset by peer (error 104) I0517 06:45:33.170545 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 14427 W0517 06:45:33.192382 14426 connection.cc:512] client connection to 127.4.96.62:56237 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:56237): Connection reset by peer (error 104) [ OK ] AdminCliTest.TestListTables (2683 ms) [ RUN ] AdminCliTest.TestListTablesDetail I0517 06:45:33.203996 4480 test_util.cc:200] Using random seed: 782023170 I0517 06:45:33.207767 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:45:33.207901 4480 ts_itest-base.cc:114] -------------- I0517 06:45:33.208004 4480 ts_itest-base.cc:115] 3 tablet servers I0517 06:45:33.208106 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:45:33.208204 4480 ts_itest-base.cc:117] -------------- I0517 06:45:33.210367 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:51382 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:33.324226 14654 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:33.324530 14654 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:33.333693 14654 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:45:33.333811 14654 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:33.333909 14654 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:45:33.334038 14654 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:45:33.348043 14654 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:51382 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:33.355604 14654 master_main.cc:80] Initializing master server... I0517 06:45:33.357081 14654 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 165819us I0517 06:45:33.357803 14654 fs_manager.cc:260] Metadata directory not provided I0517 06:45:33.357939 14654 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:33.358139 14654 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:45:33.358252 14654 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:33.365329 14654 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "11130566570c4bc9a9ced0d3db2066a3" format_stamp: "Formatted at 2018-05-17 06:45:33 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:33.365965 14654 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "11130566570c4bc9a9ced0d3db2066a3" format_stamp: "Formatted at 2018-05-17 06:45:33 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:33.375079 14654 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:45:33.376173 14654 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:33.376556 14654 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:33.382879 14654 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:45:33.382984 14654 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "11130566570c4bc9a9ced0d3db2066a3" format_stamp: "Formatted at 2018-05-17 06:45:33 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:33.383174 14654 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:33.533747 14654 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:33.535434 14654 master_main.cc:83] Starting Master server... I0517 06:45:33.581323 14654 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:51382 I0517 06:45:33.582201 14654 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:45:33.582307 14654 webserver.cc:184] Document root disabled I0517 06:45:33.583520 14654 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:46693/ I0517 06:45:33.584239 14654 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:45:33.585321 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 14654 I0517 06:45:33.586722 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:51382 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:33.587311 14715 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:33.602538 14715 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3: Bootstrap starting. I0517 06:45:33.604688 14715 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3: No blocks or log segments found. Creating new log. I0517 06:45:33.605810 14715 log.cc:520] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3: Log is configured to *not* fsync() on all Append() calls I0517 06:45:33.608163 14715 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3: No bootstrap required, opened a new log I0517 06:45:33.619696 14715 raft_consensus.cc:303] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11130566570c4bc9a9ced0d3db2066a3" member_type: VOTER } I0517 06:45:33.620002 14715 raft_consensus.cc:329] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:33.620136 14715 raft_consensus.cc:604] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11130566570c4bc9a9ced0d3db2066a3, State: Initialized, Role: FOLLOWER I0517 06:45:33.620530 14715 consensus_queue.cc:227] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11130566570c4bc9a9ced0d3db2066a3" member_type: VOTER } I0517 06:45:33.620839 14715 raft_consensus.cc:343] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:45:33.620973 14715 raft_consensus.cc:435] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:45:33.621120 14715 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3: Snoozing failure detection for 1.580s (starting election) I0517 06:45:33.621258 14715 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:33.623625 14715 raft_consensus.cc:457] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11130566570c4bc9a9ced0d3db2066a3" member_type: VOTER } I0517 06:45:33.624038 14715 leader_election.cc:270] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:33.625624 14718 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3: Snoozing failure detection for 1.877s (election complete) I0517 06:45:33.625910 14718 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:33.638438 14718 raft_consensus.cc:571] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [term 1 LEADER]: Becoming Leader. State: Replica: 11130566570c4bc9a9ced0d3db2066a3, State: Running, Role: LEADER I0517 06:45:33.638823 14718 consensus_queue.cc:204] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11130566570c4bc9a9ced0d3db2066a3" member_type: VOTER } I0517 06:45:33.642009 14715 sys_catalog.cc:335] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "11130566570c4bc9a9ced0d3db2066a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11130566570c4bc9a9ced0d3db2066a3" member_type: VOTER } } I0517 06:45:33.642457 14715 sys_catalog.cc:338] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [sys.catalog]: This master's current role is: LEADER I0517 06:45:33.645937 14722 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:45:33.645166 14719 sys_catalog.cc:335] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11130566570c4bc9a9ced0d3db2066a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11130566570c4bc9a9ced0d3db2066a3" member_type: VOTER } } I0517 06:45:33.706681 14719 sys_catalog.cc:338] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [sys.catalog]: This master's current role is: LEADER I0517 06:45:33.645232 14720 sys_catalog.cc:335] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11130566570c4bc9a9ced0d3db2066a3. Latest consensus state: current_term: 1 leader_uuid: "11130566570c4bc9a9ced0d3db2066a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11130566570c4bc9a9ced0d3db2066a3" member_type: VOTER } } I0517 06:45:33.708845 14720 sys_catalog.cc:338] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [sys.catalog]: This master's current role is: LEADER I0517 06:45:33.649629 14715 sys_catalog.cc:422] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:45:33.714798 14722 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:45:33.726819 14654 master_main.cc:86] Master server successfully started. WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:33.728551 14716 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:33.729033 14716 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:33.743244 14716 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:33.743744 14716 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:45:33.756018 14722 catalog_manager.cc:899] Generated new certificate authority record I0517 06:45:33.758350 14722 catalog_manager.cc:1021] Loading token signing keys... I0517 06:45:33.770334 14722 catalog_manager.cc:3962] T 00000000000000000000000000000000 P 11130566570c4bc9a9ced0d3db2066a3: Generated new TSK 0 I0517 06:45:33.770555 14716 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:51382 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:33.778765 14716 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:33.780573 14716 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 165819us I0517 06:45:33.781005 14716 fs_manager.cc:260] Metadata directory not provided I0517 06:45:33.781131 14716 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:33.781332 14716 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:45:33.781443 14716 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:33.789290 14716 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "7f5cb95271f04f4d8fad2f9f00f17396" format_stamp: "Formatted at 2018-05-17 06:45:33 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:33.789968 14716 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "7f5cb95271f04f4d8fad2f9f00f17396" format_stamp: "Formatted at 2018-05-17 06:45:33 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:33.819098 14716 fs_manager.cc:495] Time spent creating directory manager: real 0.029s user 0.004s sys 0.024s I0517 06:45:33.820235 14716 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:33.820641 14716 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:33.831109 14716 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.004s sys 0.008s I0517 06:45:33.831221 14716 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "7f5cb95271f04f4d8fad2f9f00f17396" format_stamp: "Formatted at 2018-05-17 06:45:33 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:33.831419 14716 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:34.078899 14716 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:34.081596 14716 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:34.094247 14716 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:34.441195 14716 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:45590 I0517 06:45:34.442145 14716 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:45:34.442250 14716 webserver.cc:184] Document root disabled I0517 06:45:34.443631 14716 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:41360/ I0517 06:45:34.444335 14716 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:45:34.447938 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 14716 I0517 06:45:34.449257 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:51382 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:34.456790 14716 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:34.475214 14847 heartbeater.cc:346] Connected to a master server at 127.4.96.62:51382 I0517 06:45:34.475471 14847 heartbeater.cc:426] Registering TS with master... I0517 06:45:34.475922 14847 heartbeater.cc:475] Master 127.4.96.62:51382 requested a full tablet report, sending... I0517 06:45:34.477449 14682 ts_manager.cc:81] Registered new tserver with Master: 7f5cb95271f04f4d8fad2f9f00f17396 (127.4.96.1:45590) I0517 06:45:34.479166 14682 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:39783 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:34.606796 14851 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:34.607267 14851 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:34.621513 14851 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:34.622007 14851 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:45:34.642127 14851 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:51382 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:34.655798 14851 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:34.658113 14851 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 166319us I0517 06:45:34.658691 14851 fs_manager.cc:260] Metadata directory not provided I0517 06:45:34.658862 14851 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:34.659118 14851 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:45:34.659271 14851 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:34.671670 14851 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "6273e1a98ee74e2299e022c4b2bf1412" format_stamp: "Formatted at 2018-05-17 06:45:34 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:34.672574 14851 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "6273e1a98ee74e2299e022c4b2bf1412" format_stamp: "Formatted at 2018-05-17 06:45:34 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:34.686287 14851 fs_manager.cc:495] Time spent creating directory manager: real 0.013s user 0.016s sys 0.000s I0517 06:45:34.687918 14851 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:34.688457 14851 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:34.699494 14851 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.004s sys 0.004s I0517 06:45:34.699646 14851 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "6273e1a98ee74e2299e022c4b2bf1412" format_stamp: "Formatted at 2018-05-17 06:45:34 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:34.699928 14851 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:34.884922 14851 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:34.888159 14851 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:34.889271 14851 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:35.051133 14851 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:57948 I0517 06:45:35.052436 14851 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:45:35.052579 14851 webserver.cc:184] Document root disabled I0517 06:45:35.054050 14851 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:49045/ I0517 06:45:35.054913 14851 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:45:35.059017 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 14851 I0517 06:45:35.060390 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:51382 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0517 06:45:35.066766 14851 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:35.089056 14973 heartbeater.cc:346] Connected to a master server at 127.4.96.62:51382 I0517 06:45:35.089294 14973 heartbeater.cc:426] Registering TS with master... I0517 06:45:35.089818 14973 heartbeater.cc:475] Master 127.4.96.62:51382 requested a full tablet report, sending... I0517 06:45:35.091825 14682 ts_manager.cc:81] Registered new tserver with Master: 6273e1a98ee74e2299e022c4b2bf1412 (127.4.96.2:57948) I0517 06:45:35.093906 14682 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:43803 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:35.169226 14977 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:35.169549 14977 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:35.177569 14977 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:35.178000 14977 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:45:35.190234 14977 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:51382 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:35.198293 14977 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:35.200047 14977 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 166819us I0517 06:45:35.200465 14977 fs_manager.cc:260] Metadata directory not provided I0517 06:45:35.200575 14977 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:35.200754 14977 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:45:35.200857 14977 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:35.208444 14977 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "a8c994bd452a4c8f9231e25b0de809d7" format_stamp: "Formatted at 2018-05-17 06:45:35 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:35.209107 14977 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "a8c994bd452a4c8f9231e25b0de809d7" format_stamp: "Formatted at 2018-05-17 06:45:35 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:35.218487 14977 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:45:35.219564 14977 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:35.219936 14977 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:35.226918 14977 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0517 06:45:35.227030 14977 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "a8c994bd452a4c8f9231e25b0de809d7" format_stamp: "Formatted at 2018-05-17 06:45:35 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:35.227259 14977 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:35.391005 14977 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:35.393332 14977 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:35.394268 14977 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:35.479610 14977 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:47487 I0517 06:45:35.480505 14977 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:45:35.480612 14977 webserver.cc:184] Document root disabled I0517 06:45:35.481554 14847 heartbeater.cc:467] Master 127.4.96.62:51382 was elected leader, sending a full tablet report... I0517 06:45:35.481855 14977 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:56277/ I0517 06:45:35.482576 14977 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:45:35.485466 14977 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:35.491688 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 14977 I0517 06:45:35.507357 15098 heartbeater.cc:346] Connected to a master server at 127.4.96.62:51382 I0517 06:45:35.507580 15098 heartbeater.cc:426] Registering TS with master... I0517 06:45:35.507983 15098 heartbeater.cc:475] Master 127.4.96.62:51382 requested a full tablet report, sending... I0517 06:45:35.509320 14682 ts_manager.cc:81] Registered new tserver with Master: a8c994bd452a4c8f9231e25b0de809d7 (127.4.96.3:47487) I0517 06:45:35.510974 14682 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:51646 I0517 06:45:35.515781 4480 external_mini_cluster.cc:482] 3 TS(s) registered with all masters I0517 06:45:35.540377 14682 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32964: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0517 06:45:35.548825 14682 catalog_manager.cc:1463] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0517 06:45:35.581985 14784 tablet_service.cc:759] Processing CreateTablet for tablet 74988dfa0c9d440d8efa3000f3148d38 (table=TestTable [id=8f37ef1f2373457f9e278757330950e4]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:35.582703 14784 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 74988dfa0c9d440d8efa3000f3148d38. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:35.590126 15035 tablet_service.cc:759] Processing CreateTablet for tablet 74988dfa0c9d440d8efa3000f3148d38 (table=TestTable [id=8f37ef1f2373457f9e278757330950e4]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:35.590934 15035 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 74988dfa0c9d440d8efa3000f3148d38. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:35.596693 14784 ts_tablet_manager.cc:1173] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:35.598240 15118 ts_tablet_manager.cc:1028] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396: Bootstrapping tablet I0517 06:45:35.598609 15118 tablet_bootstrap.cc:436] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396: Bootstrap starting. I0517 06:45:35.599855 14910 tablet_service.cc:759] Processing CreateTablet for tablet 74988dfa0c9d440d8efa3000f3148d38 (table=TestTable [id=8f37ef1f2373457f9e278757330950e4]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:35.600448 14910 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 74988dfa0c9d440d8efa3000f3148d38. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:35.601336 15118 tablet_bootstrap.cc:587] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396: No blocks or log segments found. Creating new log. I0517 06:45:35.602695 15118 log.cc:520] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396: Log is configured to *not* fsync() on all Append() calls I0517 06:45:35.615721 15035 ts_tablet_manager.cc:1173] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:35.617440 14910 ts_tablet_manager.cc:1173] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:35.618605 15120 ts_tablet_manager.cc:1028] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7: Bootstrapping tablet I0517 06:45:35.619088 15120 tablet_bootstrap.cc:436] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7: Bootstrap starting. I0517 06:45:35.619909 15121 ts_tablet_manager.cc:1028] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412: Bootstrapping tablet I0517 06:45:35.620625 15121 tablet_bootstrap.cc:436] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412: Bootstrap starting. I0517 06:45:35.621568 15118 tablet_bootstrap.cc:436] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396: No bootstrap required, opened a new log I0517 06:45:35.621917 15120 tablet_bootstrap.cc:587] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7: No blocks or log segments found. Creating new log. I0517 06:45:35.621969 15118 ts_tablet_manager.cc:1045] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.012s I0517 06:45:35.623345 15121 tablet_bootstrap.cc:587] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412: No blocks or log segments found. Creating new log. I0517 06:45:35.623481 15120 log.cc:520] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7: Log is configured to *not* fsync() on all Append() calls I0517 06:45:35.624480 15121 log.cc:520] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412: Log is configured to *not* fsync() on all Append() calls I0517 06:45:35.629488 15120 tablet_bootstrap.cc:436] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7: No bootstrap required, opened a new log I0517 06:45:35.629839 15120 ts_tablet_manager.cc:1045] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s I0517 06:45:35.637063 15121 tablet_bootstrap.cc:436] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412: No bootstrap required, opened a new log I0517 06:45:35.637439 15121 ts_tablet_manager.cc:1045] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.008s I0517 06:45:35.638098 15118 raft_consensus.cc:303] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } I0517 06:45:35.638916 15118 raft_consensus.cc:329] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:35.639138 15118 raft_consensus.cc:604] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5cb95271f04f4d8fad2f9f00f17396, State: Initialized, Role: FOLLOWER I0517 06:45:35.639871 15118 consensus_queue.cc:227] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } I0517 06:45:35.641033 15120 raft_consensus.cc:303] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } I0517 06:45:35.641719 15120 raft_consensus.cc:329] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:35.641899 15120 raft_consensus.cc:604] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c994bd452a4c8f9231e25b0de809d7, State: Initialized, Role: FOLLOWER I0517 06:45:35.642498 15120 consensus_queue.cc:227] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } I0517 06:45:35.649602 15118 ts_tablet_manager.cc:1073] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396: Time spent starting tablet: real 0.027s user 0.012s sys 0.008s I0517 06:45:35.650631 15098 heartbeater.cc:467] Master 127.4.96.62:51382 was elected leader, sending a full tablet report... I0517 06:45:35.652755 15121 raft_consensus.cc:303] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } I0517 06:45:35.653445 15121 raft_consensus.cc:329] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:35.653666 15121 raft_consensus.cc:604] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6273e1a98ee74e2299e022c4b2bf1412, State: Initialized, Role: FOLLOWER I0517 06:45:35.655405 15121 consensus_queue.cc:227] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } I0517 06:45:35.656188 15126 raft_consensus.cc:435] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:35.656443 15126 raft_consensus.cc:2704] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412: Snoozing failure detection for 1.656s (starting election) I0517 06:45:35.656707 15126 raft_consensus.cc:457] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } I0517 06:45:35.657999 15126 leader_election.cc:243] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a8c994bd452a4c8f9231e25b0de809d7 (127.4.96.3:47487) I0517 06:45:35.659816 15126 leader_election.cc:243] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 7f5cb95271f04f4d8fad2f9f00f17396 (127.4.96.1:45590) I0517 06:45:35.650993 15120 ts_tablet_manager.cc:1073] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s I0517 06:45:35.657424 14973 heartbeater.cc:467] Master 127.4.96.62:51382 was elected leader, sending a full tablet report... I0517 06:45:35.729382 15055 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "74988dfa0c9d440d8efa3000f3148d38" candidate_uuid: "6273e1a98ee74e2299e022c4b2bf1412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c994bd452a4c8f9231e25b0de809d7" is_pre_election: true I0517 06:45:35.730072 15055 raft_consensus.cc:2704] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7: Snoozing failure detection for 2.066s (vote granted) I0517 06:45:35.730322 15055 raft_consensus.cc:2219] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6273e1a98ee74e2299e022c4b2bf1412 in term 0. I0517 06:45:35.731539 14862 leader_election.cc:379] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a8c994bd452a4c8f9231e25b0de809d7 (127.4.96.3:47487) I0517 06:45:35.731793 14862 leader_election.cc:270] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:45:35.732362 15126 raft_consensus.cc:2704] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412: Snoozing failure detection for 2.114s (election complete) I0517 06:45:35.732580 15126 raft_consensus.cc:2511] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:45:35.732762 15126 raft_consensus.cc:435] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:45:35.732923 15126 raft_consensus.cc:2704] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412: Snoozing failure detection for 1.816s (starting election) I0517 06:45:35.733109 15126 raft_consensus.cc:2756] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [term 0 FOLLOWER]: Advancing to term 1 W0517 06:45:35.735555 15099 tablet.cc:1678] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:35.737293 15126 raft_consensus.cc:457] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } I0517 06:45:35.738229 15126 leader_election.cc:243] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [CANDIDATE]: Term 1 election: Requesting vote from peer a8c994bd452a4c8f9231e25b0de809d7 (127.4.96.3:47487) I0517 06:45:35.739418 15055 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "74988dfa0c9d440d8efa3000f3148d38" candidate_uuid: "6273e1a98ee74e2299e022c4b2bf1412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c994bd452a4c8f9231e25b0de809d7" I0517 06:45:35.739814 15055 raft_consensus.cc:2756] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:35.740072 15055 raft_consensus.cc:2704] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7: Snoozing failure detection for 2.224s (vote granted) I0517 06:45:35.744451 15126 leader_election.cc:243] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [CANDIDATE]: Term 1 election: Requesting vote from peer 7f5cb95271f04f4d8fad2f9f00f17396 (127.4.96.1:45590) I0517 06:45:35.744251 15055 raft_consensus.cc:2219] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6273e1a98ee74e2299e022c4b2bf1412 in term 1. I0517 06:45:35.745687 14862 leader_election.cc:379] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [CANDIDATE]: Term 1 election: Vote granted by peer a8c994bd452a4c8f9231e25b0de809d7 (127.4.96.3:47487) I0517 06:45:35.745920 14862 leader_election.cc:270] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:35.746443 15126 raft_consensus.cc:2704] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412: Snoozing failure detection for 1.710s (election complete) I0517 06:45:35.746675 15126 raft_consensus.cc:2511] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:35.663481 15121 ts_tablet_manager.cc:1073] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412: Time spent starting tablet: real 0.026s user 0.012s sys 0.008s I0517 06:45:35.747104 15126 raft_consensus.cc:571] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [term 1 LEADER]: Becoming Leader. State: Replica: 6273e1a98ee74e2299e022c4b2bf1412, State: Running, Role: LEADER I0517 06:45:35.747555 15126 consensus_queue.cc:204] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } I0517 06:45:35.759902 14804 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "74988dfa0c9d440d8efa3000f3148d38" candidate_uuid: "6273e1a98ee74e2299e022c4b2bf1412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" is_pre_election: true I0517 06:45:35.759630 14681 catalog_manager.cc:3686] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 reported cstate change: term changed from 0 to 1, leader changed from to 6273e1a98ee74e2299e022c4b2bf1412 (127.4.96.2). New cstate: current_term: 1 leader_uuid: "6273e1a98ee74e2299e022c4b2bf1412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } health_report { overall_health: HEALTHY } } } I0517 06:45:35.760469 14804 raft_consensus.cc:2704] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396: Snoozing failure detection for 1.980s (vote granted) I0517 06:45:35.760722 14804 raft_consensus.cc:2219] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6273e1a98ee74e2299e022c4b2bf1412 in term 0. I0517 06:45:35.764596 14861 leader_election.cc:379] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 7f5cb95271f04f4d8fad2f9f00f17396 (127.4.96.1:45590) I0517 06:45:35.769731 4480 external_mini_cluster.cc:482] 3 TS(s) registered with all masters I0517 06:45:35.771761 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7f5cb95271f04f4d8fad2f9f00f17396 to finish bootstrapping I0517 06:45:35.781765 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6273e1a98ee74e2299e022c4b2bf1412 to finish bootstrapping W0517 06:45:35.707412 14848 tablet.cc:1678] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:35.791214 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a8c994bd452a4c8f9231e25b0de809d7 to finish bootstrapping I0517 06:45:35.801900 14681 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32964: name: "TestAnotherTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0517 06:45:35.802767 14681 catalog_manager.cc:1463] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0517 06:45:35.761626 14803 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "74988dfa0c9d440d8efa3000f3148d38" candidate_uuid: "6273e1a98ee74e2299e022c4b2bf1412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" I0517 06:45:35.806442 14803 raft_consensus.cc:2756] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:35.806658 14803 raft_consensus.cc:2704] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396: Snoozing failure detection for 2.079s (vote granted) I0517 06:45:35.810367 14803 raft_consensus.cc:2219] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6273e1a98ee74e2299e022c4b2bf1412 in term 1. I0517 06:45:35.811553 14784 tablet_service.cc:759] Processing CreateTablet for tablet 4f7b792e38a74647a88f4a6e60394e61 (table=TestAnotherTable [id=3c90ebc0e0c54e32ad79136082b06516]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:35.811689 14910 tablet_service.cc:759] Processing CreateTablet for tablet 4f7b792e38a74647a88f4a6e60394e61 (table=TestAnotherTable [id=3c90ebc0e0c54e32ad79136082b06516]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:35.812117 14784 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4f7b792e38a74647a88f4a6e60394e61. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:35.812104 15035 tablet_service.cc:759] Processing CreateTablet for tablet 4f7b792e38a74647a88f4a6e60394e61 (table=TestAnotherTable [id=3c90ebc0e0c54e32ad79136082b06516]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:35.812283 14910 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4f7b792e38a74647a88f4a6e60394e61. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:35.812675 15035 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4f7b792e38a74647a88f4a6e60394e61. 1 dirs total, 0 dirs full, 0 dirs failed W0517 06:45:35.817068 14974 tablet.cc:1678] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:35.817267 14784 ts_tablet_manager.cc:1173] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:35.817570 15118 ts_tablet_manager.cc:1028] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396: Bootstrapping tablet I0517 06:45:35.817905 15118 tablet_bootstrap.cc:436] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396: Bootstrap starting. I0517 06:45:35.819880 15118 tablet_bootstrap.cc:587] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396: No blocks or log segments found. Creating new log. I0517 06:45:35.821053 15035 ts_tablet_manager.cc:1173] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:35.821061 14910 ts_tablet_manager.cc:1173] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:35.821395 15120 ts_tablet_manager.cc:1028] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7: Bootstrapping tablet I0517 06:45:35.821615 14861 leader_election.cc:379] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [CANDIDATE]: Term 1 election: Vote granted by peer 7f5cb95271f04f4d8fad2f9f00f17396 (127.4.96.1:45590) I0517 06:45:35.821734 15120 tablet_bootstrap.cc:436] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7: Bootstrap starting. I0517 06:45:35.822127 15121 ts_tablet_manager.cc:1028] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412: Bootstrapping tablet I0517 06:45:35.822455 15121 tablet_bootstrap.cc:436] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412: Bootstrap starting. I0517 06:45:35.824004 15120 tablet_bootstrap.cc:587] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7: No blocks or log segments found. Creating new log. I0517 06:45:35.824801 15121 tablet_bootstrap.cc:587] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412: No blocks or log segments found. Creating new log. I0517 06:45:35.828615 15121 tablet_bootstrap.cc:436] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412: No bootstrap required, opened a new log I0517 06:45:35.828938 15121 ts_tablet_manager.cc:1045] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s I0517 06:45:35.829030 15118 tablet_bootstrap.cc:436] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396: No bootstrap required, opened a new log I0517 06:45:35.829254 15118 ts_tablet_manager.cc:1045] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396: Time spent bootstrapping tablet: real 0.011s user 0.012s sys 0.000s I0517 06:45:35.830185 15118 raft_consensus.cc:303] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } I0517 06:45:35.830260 15121 raft_consensus.cc:303] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } I0517 06:45:35.830600 15118 raft_consensus.cc:329] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:35.830651 15121 raft_consensus.cc:329] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:35.830761 15121 raft_consensus.cc:604] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6273e1a98ee74e2299e022c4b2bf1412, State: Initialized, Role: FOLLOWER I0517 06:45:35.830747 15118 raft_consensus.cc:604] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5cb95271f04f4d8fad2f9f00f17396, State: Initialized, Role: FOLLOWER I0517 06:45:35.831068 15121 consensus_queue.cc:227] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } I0517 06:45:35.831125 15118 consensus_queue.cc:227] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } I0517 06:45:35.832116 15121 ts_tablet_manager.cc:1073] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s I0517 06:45:35.832556 15118 ts_tablet_manager.cc:1073] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s I0517 06:45:35.833057 15120 tablet_bootstrap.cc:436] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7: No bootstrap required, opened a new log I0517 06:45:35.833349 15120 ts_tablet_manager.cc:1045] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s I0517 06:45:35.834688 15120 raft_consensus.cc:303] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } I0517 06:45:35.835281 15120 raft_consensus.cc:329] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:35.835474 15120 raft_consensus.cc:604] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c994bd452a4c8f9231e25b0de809d7, State: Initialized, Role: FOLLOWER I0517 06:45:35.835976 15120 consensus_queue.cc:227] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } I0517 06:45:35.837182 15120 ts_tablet_manager.cc:1073] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s I0517 06:45:36.020812 15125 raft_consensus.cc:435] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:36.021028 15125 raft_consensus.cc:2704] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7: Snoozing failure detection for 1.905s (starting election) I0517 06:45:36.021186 15125 raft_consensus.cc:457] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } I0517 06:45:36.022054 15125 leader_election.cc:243] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 7f5cb95271f04f4d8fad2f9f00f17396 (127.4.96.1:45590) I0517 06:45:36.022441 15125 leader_election.cc:243] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6273e1a98ee74e2299e022c4b2bf1412 (127.4.96.2:57948) I0517 06:45:36.033308 14803 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4f7b792e38a74647a88f4a6e60394e61" candidate_uuid: "a8c994bd452a4c8f9231e25b0de809d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" is_pre_election: true I0517 06:45:36.033710 14803 raft_consensus.cc:2704] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396: Snoozing failure detection for 1.552s (vote granted) I0517 06:45:36.033893 14803 raft_consensus.cc:2219] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c994bd452a4c8f9231e25b0de809d7 in term 0. I0517 06:45:36.034170 14930 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4f7b792e38a74647a88f4a6e60394e61" candidate_uuid: "a8c994bd452a4c8f9231e25b0de809d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6273e1a98ee74e2299e022c4b2bf1412" is_pre_election: true I0517 06:45:36.034703 14930 raft_consensus.cc:2704] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412: Snoozing failure detection for 1.847s (vote granted) I0517 06:45:36.034744 14986 leader_election.cc:379] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 7f5cb95271f04f4d8fad2f9f00f17396 (127.4.96.1:45590) I0517 06:45:36.034971 14986 leader_election.cc:270] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:45:36.034932 14930 raft_consensus.cc:2219] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c994bd452a4c8f9231e25b0de809d7 in term 0. I0517 06:45:36.035457 15125 raft_consensus.cc:2704] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7: Snoozing failure detection for 1.829s (election complete) I0517 06:45:36.035686 15125 raft_consensus.cc:2511] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:45:36.035825 15125 raft_consensus.cc:435] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:45:36.035846 14987 leader_election.cc:379] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 6273e1a98ee74e2299e022c4b2bf1412 (127.4.96.2:57948) I0517 06:45:36.035964 15125 raft_consensus.cc:2704] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7: Snoozing failure detection for 1.780s (starting election) I0517 06:45:36.036090 15125 raft_consensus.cc:2756] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:36.038988 15125 raft_consensus.cc:457] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } I0517 06:45:36.039590 15125 leader_election.cc:243] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [CANDIDATE]: Term 1 election: Requesting vote from peer 7f5cb95271f04f4d8fad2f9f00f17396 (127.4.96.1:45590) I0517 06:45:36.039852 15125 leader_election.cc:243] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [CANDIDATE]: Term 1 election: Requesting vote from peer 6273e1a98ee74e2299e022c4b2bf1412 (127.4.96.2:57948) I0517 06:45:36.040717 14803 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4f7b792e38a74647a88f4a6e60394e61" candidate_uuid: "a8c994bd452a4c8f9231e25b0de809d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" I0517 06:45:36.040856 14930 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4f7b792e38a74647a88f4a6e60394e61" candidate_uuid: "a8c994bd452a4c8f9231e25b0de809d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6273e1a98ee74e2299e022c4b2bf1412" I0517 06:45:36.041128 14803 raft_consensus.cc:2756] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:36.041236 14930 raft_consensus.cc:2756] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:36.041344 14803 raft_consensus.cc:2704] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396: Snoozing failure detection for 1.913s (vote granted) I0517 06:45:36.041452 14930 raft_consensus.cc:2704] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412: Snoozing failure detection for 2.144s (vote granted) I0517 06:45:36.045354 14930 raft_consensus.cc:2219] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c994bd452a4c8f9231e25b0de809d7 in term 1. I0517 06:45:36.045372 14803 raft_consensus.cc:2219] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c994bd452a4c8f9231e25b0de809d7 in term 1. I0517 06:45:36.046037 14986 leader_election.cc:379] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [CANDIDATE]: Term 1 election: Vote granted by peer 7f5cb95271f04f4d8fad2f9f00f17396 (127.4.96.1:45590) I0517 06:45:36.046206 14986 leader_election.cc:270] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:36.046375 14987 leader_election.cc:379] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [CANDIDATE]: Term 1 election: Vote granted by peer 6273e1a98ee74e2299e022c4b2bf1412 (127.4.96.2:57948) I0517 06:45:36.046530 15125 raft_consensus.cc:2704] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7: Snoozing failure detection for 1.643s (election complete) I0517 06:45:36.046744 15125 raft_consensus.cc:2511] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:36.047745 15125 raft_consensus.cc:571] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [term 1 LEADER]: Becoming Leader. State: Replica: a8c994bd452a4c8f9231e25b0de809d7, State: Running, Role: LEADER I0517 06:45:36.048099 15125 consensus_queue.cc:204] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } } peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } } I0517 06:45:36.052837 14681 catalog_manager.cc:3686] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 reported cstate change: term changed from 0 to 1, leader changed from to a8c994bd452a4c8f9231e25b0de809d7 (127.4.96.3). New cstate: current_term: 1 leader_uuid: "a8c994bd452a4c8f9231e25b0de809d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 } health_report { overall_health: HEALTHY } } } I0517 06:45:36.153293 15055 raft_consensus.cc:1057] T 74988dfa0c9d440d8efa3000f3148d38 P a8c994bd452a4c8f9231e25b0de809d7 [term 1 FOLLOWER]: Refusing update from remote peer 6273e1a98ee74e2299e022c4b2bf1412: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:45:36.154510 15126 consensus_queue.cc:963] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c994bd452a4c8f9231e25b0de809d7" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:36.161403 14803 raft_consensus.cc:1057] T 74988dfa0c9d440d8efa3000f3148d38 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 1 FOLLOWER]: Refusing update from remote peer 6273e1a98ee74e2299e022c4b2bf1412: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:45:36.162817 15127 consensus_queue.cc:963] T 74988dfa0c9d440d8efa3000f3148d38 P 6273e1a98ee74e2299e022c4b2bf1412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0517 06:45:36.193452 15139 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:45:36.448747 14930 raft_consensus.cc:1057] T 4f7b792e38a74647a88f4a6e60394e61 P 6273e1a98ee74e2299e022c4b2bf1412 [term 1 FOLLOWER]: Refusing update from remote peer a8c994bd452a4c8f9231e25b0de809d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:45:36.449791 15125 consensus_queue.cc:963] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6273e1a98ee74e2299e022c4b2bf1412" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 57948 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:36.458746 14803 raft_consensus.cc:1057] T 4f7b792e38a74647a88f4a6e60394e61 P 7f5cb95271f04f4d8fad2f9f00f17396 [term 1 FOLLOWER]: Refusing update from remote peer a8c994bd452a4c8f9231e25b0de809d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0517 06:45:36.459658 15152 consensus_queue.cc:963] T 4f7b792e38a74647a88f4a6e60394e61 P a8c994bd452a4c8f9231e25b0de809d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f5cb95271f04f4d8fad2f9f00f17396" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 45590 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:37.425462 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 14716 W0517 06:45:37.445960 14663 connection.cc:512] client connection to 127.4.96.1:45590 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:45590): Connection reset by peer (error 104) W0517 06:45:37.446199 14861 connection.cc:512] client connection to 127.4.96.1:45590 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:45590): Connection reset by peer (error 104) W0517 06:45:37.446265 14653 connection.cc:512] client connection to 127.4.96.1:45590 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:45590): Connection reset by peer (error 104) W0517 06:45:37.446897 14666 connection.cc:512] server connection from 127.4.96.1:39783 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:39783): Connection reset by peer (error 104) W0517 06:45:37.447366 14986 connection.cc:512] client connection to 127.4.96.1:45590 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:45590): Connection reset by peer (error 104) W0517 06:45:37.448299 14650 connection.cc:512] client connection to 127.4.96.1:45590 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:45590): Connection reset by peer (error 104) I0517 06:45:37.448380 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 14851 W0517 06:45:37.476469 14664 connection.cc:512] client connection to 127.4.96.2:57948 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:57948): Connection reset by peer (error 104) W0517 06:45:37.476510 14985 connection.cc:512] server connection from 127.4.96.2:39289 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:39289): Connection reset by peer (error 104) W0517 06:45:37.477123 14987 connection.cc:512] client connection to 127.4.96.2:57948 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:57948): Connection reset by peer (error 104) W0517 06:45:37.477253 14664 connection.cc:512] server connection from 127.4.96.2:43803 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:43803): Connection reset by peer (error 104) I0517 06:45:37.477800 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 14977 W0517 06:45:37.479243 14653 connection.cc:512] client connection to 127.4.96.2:57948 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:57948): Connection reset by peer (error 104) W0517 06:45:37.504739 14664 connection.cc:512] client connection to 127.4.96.3:47487 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:47487): Connection reset by peer (error 104) W0517 06:45:37.505400 14653 connection.cc:512] client connection to 127.4.96.3:47487 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:47487): Connection reset by peer (error 104) I0517 06:45:37.506042 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 14654 W0517 06:45:37.532232 14653 connection.cc:512] client connection to 127.4.96.62:51382 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:51382): Connection reset by peer (error 104) [ OK ] AdminCliTest.TestListTablesDetail (4342 ms) [----------] 14 tests from AdminCliTest (103131 ms total) [----------] 6 tests from EnableKudu1097AndDownTS/MoveTabletParamTest [ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/0 I0517 06:45:37.545341 4480 test_util.cc:200] Using random seed: 786364517 I0517 06:45:37.548575 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:45:37.548679 4480 ts_itest-base.cc:114] -------------- I0517 06:45:37.548766 4480 ts_itest-base.cc:115] 5 tablet servers I0517 06:45:37.548851 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:45:37.548936 4480 ts_itest-base.cc:117] -------------- I0517 06:45:37.551095 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:45644 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:37.664270 15163 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:37.664598 15163 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:37.673521 15163 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:37.673688 15163 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:45:37.673800 15163 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:37.673907 15163 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:45:37.674052 15163 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:45:37.687955 15163 master_main.cc:73] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:45644 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:37.695578 15163 master_main.cc:80] Initializing master server... I0517 06:45:37.697079 15163 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 167819us I0517 06:45:37.697805 15163 fs_manager.cc:260] Metadata directory not provided I0517 06:45:37.697945 15163 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:37.698160 15163 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:45:37.698277 15163 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:37.705401 15163 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "fc6df897b46e454f8b2e219dc5f39d44" format_stamp: "Formatted at 2018-05-17 06:45:37 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:37.706089 15163 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "fc6df897b46e454f8b2e219dc5f39d44" format_stamp: "Formatted at 2018-05-17 06:45:37 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:37.715231 15163 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:45:37.716327 15163 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:37.716709 15163 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:37.723019 15163 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.004s I0517 06:45:37.723155 15163 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "fc6df897b46e454f8b2e219dc5f39d44" format_stamp: "Formatted at 2018-05-17 06:45:37 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:37.723359 15163 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:37.892951 15163 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:37.894657 15163 master_main.cc:83] Starting Master server... I0517 06:45:37.933769 15163 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:45644 I0517 06:45:37.934633 15163 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:45:37.934742 15163 webserver.cc:184] Document root disabled I0517 06:45:37.935981 15163 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:59623/ I0517 06:45:37.936703 15163 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:45:37.939254 15224 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:37.943253 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 15163 I0517 06:45:37.944218 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:45644 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:37.952905 15224 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44: Bootstrap starting. I0517 06:45:37.955155 15224 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44: No blocks or log segments found. Creating new log. I0517 06:45:37.956291 15224 log.cc:520] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44: Log is configured to *not* fsync() on all Append() calls I0517 06:45:37.958571 15224 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44: No bootstrap required, opened a new log I0517 06:45:37.970094 15224 raft_consensus.cc:303] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6df897b46e454f8b2e219dc5f39d44" member_type: VOTER } I0517 06:45:37.970399 15224 raft_consensus.cc:329] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:37.970526 15224 raft_consensus.cc:604] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6df897b46e454f8b2e219dc5f39d44, State: Initialized, Role: FOLLOWER I0517 06:45:37.970949 15224 consensus_queue.cc:227] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6df897b46e454f8b2e219dc5f39d44" member_type: VOTER } I0517 06:45:37.971295 15224 raft_consensus.cc:343] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:45:37.971433 15224 raft_consensus.cc:435] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:45:37.971572 15224 raft_consensus.cc:2704] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44: Snoozing failure detection for 2.054s (starting election) I0517 06:45:37.971705 15224 raft_consensus.cc:2756] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:37.974097 15224 raft_consensus.cc:457] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6df897b46e454f8b2e219dc5f39d44" member_type: VOTER } I0517 06:45:37.974509 15224 leader_election.cc:270] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:37.976769 15227 raft_consensus.cc:2704] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44: Snoozing failure detection for 1.810s (election complete) I0517 06:45:37.977036 15227 raft_consensus.cc:2511] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:37.982421 15227 raft_consensus.cc:571] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [term 1 LEADER]: Becoming Leader. State: Replica: fc6df897b46e454f8b2e219dc5f39d44, State: Running, Role: LEADER I0517 06:45:37.982828 15227 consensus_queue.cc:204] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6df897b46e454f8b2e219dc5f39d44" member_type: VOTER } I0517 06:45:37.999941 15228 sys_catalog.cc:335] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc6df897b46e454f8b2e219dc5f39d44. Latest consensus state: current_term: 1 leader_uuid: "fc6df897b46e454f8b2e219dc5f39d44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6df897b46e454f8b2e219dc5f39d44" member_type: VOTER } } I0517 06:45:38.000289 15228 sys_catalog.cc:338] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [sys.catalog]: This master's current role is: LEADER WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:38.057258 15225 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:38.057605 15225 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:45:38.001765 15229 sys_catalog.cc:335] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc6df897b46e454f8b2e219dc5f39d44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6df897b46e454f8b2e219dc5f39d44" member_type: VOTER } } I0517 06:45:38.061460 15229 sys_catalog.cc:338] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [sys.catalog]: This master's current role is: LEADER I0517 06:45:38.002507 15224 sys_catalog.cc:335] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "fc6df897b46e454f8b2e219dc5f39d44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6df897b46e454f8b2e219dc5f39d44" member_type: VOTER } } I0517 06:45:38.064473 15224 sys_catalog.cc:338] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [sys.catalog]: This master's current role is: LEADER I0517 06:45:38.065378 15224 sys_catalog.cc:422] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:45:38.065704 15231 catalog_manager.cc:1001] Loading table and tablet metadata into memory... W0517 06:45:38.066553 15225 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:38.066807 15225 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 I0517 06:45:38.067034 15163 master_main.cc:86] Master server successfully started. W0517 06:45:38.067245 15225 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:45:38.070509 15231 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:45:38.087574 15225 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:45644 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:38.100803 15225 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:38.103277 15225 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 168319us I0517 06:45:38.103917 15225 fs_manager.cc:260] Metadata directory not provided I0517 06:45:38.104085 15225 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:38.104357 15225 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:45:38.104529 15225 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:38.117455 15225 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "6af7d63e13c84ed6b72692f9c995491f" format_stamp: "Formatted at 2018-05-17 06:45:38 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:38.118438 15225 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "6af7d63e13c84ed6b72692f9c995491f" format_stamp: "Formatted at 2018-05-17 06:45:38 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:38.133019 15225 fs_manager.cc:495] Time spent creating directory manager: real 0.014s user 0.012s sys 0.004s I0517 06:45:38.135113 15225 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:38.135661 15225 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:38.155078 15225 fs_manager.cc:417] Time spent opening block manager: real 0.011s user 0.000s sys 0.012s I0517 06:45:38.155256 15225 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "6af7d63e13c84ed6b72692f9c995491f" format_stamp: "Formatted at 2018-05-17 06:45:38 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:38.155591 15225 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:38.159994 15231 catalog_manager.cc:899] Generated new certificate authority record I0517 06:45:38.161423 15231 catalog_manager.cc:1021] Loading token signing keys... I0517 06:45:38.171263 15231 catalog_manager.cc:3962] T 00000000000000000000000000000000 P fc6df897b46e454f8b2e219dc5f39d44: Generated new TSK 0 I0517 06:45:38.333083 15225 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:38.336290 15225 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:38.337440 15225 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:38.710289 15225 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:36956 I0517 06:45:38.711227 15225 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:45:38.711333 15225 webserver.cc:184] Document root disabled I0517 06:45:38.714525 15225 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:47921/ I0517 06:45:38.715198 15225 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:45:38.721472 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 15225 I0517 06:45:38.722676 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:45644 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:38.724395 15225 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:38.756381 15356 heartbeater.cc:346] Connected to a master server at 127.4.96.62:45644 I0517 06:45:38.756675 15356 heartbeater.cc:426] Registering TS with master... I0517 06:45:38.757192 15356 heartbeater.cc:475] Master 127.4.96.62:45644 requested a full tablet report, sending... I0517 06:45:38.759147 15191 ts_manager.cc:81] Registered new tserver with Master: 6af7d63e13c84ed6b72692f9c995491f (127.4.96.1:36956) I0517 06:45:38.761518 15191 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:54550 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:38.848176 15360 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:38.848487 15360 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:38.856498 15360 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:38.856653 15360 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:38.856943 15360 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:45:38.872650 15360 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:45644 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:38.881080 15360 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:38.882869 15360 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 168319us I0517 06:45:38.883311 15360 fs_manager.cc:260] Metadata directory not provided I0517 06:45:38.883432 15360 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:38.883627 15360 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:45:38.883731 15360 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:38.891352 15360 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "4f3c3b664af3423ea5b117fdb6529c92" format_stamp: "Formatted at 2018-05-17 06:45:38 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:38.892349 15360 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "4f3c3b664af3423ea5b117fdb6529c92" format_stamp: "Formatted at 2018-05-17 06:45:38 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:38.906759 15360 fs_manager.cc:495] Time spent creating directory manager: real 0.014s user 0.016s sys 0.000s I0517 06:45:38.908337 15360 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:38.908838 15360 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:38.920017 15360 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:45:38.920192 15360 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "4f3c3b664af3423ea5b117fdb6529c92" format_stamp: "Formatted at 2018-05-17 06:45:38 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:38.920482 15360 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:39.157404 15360 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:39.160027 15360 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:39.161162 15360 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:39.264936 15360 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:40737 I0517 06:45:39.265874 15360 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:45:39.266008 15360 webserver.cc:184] Document root disabled I0517 06:45:39.267395 15360 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:57772/ I0517 06:45:39.268087 15360 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:45:39.275908 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 15360 I0517 06:45:39.277207 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:45644 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:39.282467 15360 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:39.295142 15481 heartbeater.cc:346] Connected to a master server at 127.4.96.62:45644 I0517 06:45:39.295439 15481 heartbeater.cc:426] Registering TS with master... I0517 06:45:39.296010 15481 heartbeater.cc:475] Master 127.4.96.62:45644 requested a full tablet report, sending... I0517 06:45:39.297670 15191 ts_manager.cc:81] Registered new tserver with Master: 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2:40737) I0517 06:45:39.299363 15191 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:51460 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:39.386953 15485 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:39.387292 15485 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:39.395236 15485 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:39.395406 15485 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:39.395714 15485 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:45:39.408612 15485 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:45644 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:39.416868 15485 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:39.418689 15485 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 168819us I0517 06:45:39.419133 15485 fs_manager.cc:260] Metadata directory not provided I0517 06:45:39.419255 15485 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:39.419456 15485 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:45:39.419569 15485 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:39.427103 15485 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "5ce72ea2903c48f792f6c4f583dbeb50" format_stamp: "Formatted at 2018-05-17 06:45:39 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:39.427773 15485 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "5ce72ea2903c48f792f6c4f583dbeb50" format_stamp: "Formatted at 2018-05-17 06:45:39 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:39.437208 15485 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:45:39.438364 15485 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:39.438742 15485 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:39.444985 15485 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:45:39.445111 15485 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "5ce72ea2903c48f792f6c4f583dbeb50" format_stamp: "Formatted at 2018-05-17 06:45:39 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:39.445302 15485 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:39.610174 15485 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:39.612515 15485 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:39.613456 15485 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:39.697747 15485 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:49834 I0517 06:45:39.698624 15485 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:45:39.698721 15485 webserver.cc:184] Document root disabled I0517 06:45:39.699894 15485 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:36381/ I0517 06:45:39.700562 15485 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:45:39.705251 15485 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:39.705315 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 15485 I0517 06:45:39.706657 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:45644 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:39.722679 15606 heartbeater.cc:346] Connected to a master server at 127.4.96.62:45644 I0517 06:45:39.722898 15606 heartbeater.cc:426] Registering TS with master... I0517 06:45:39.723297 15606 heartbeater.cc:475] Master 127.4.96.62:45644 requested a full tablet report, sending... I0517 06:45:39.724561 15191 ts_manager.cc:81] Registered new tserver with Master: 5ce72ea2903c48f792f6c4f583dbeb50 (127.4.96.3:49834) I0517 06:45:39.725622 15191 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:35771 I0517 06:45:39.764494 15356 heartbeater.cc:467] Master 127.4.96.62:45644 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:39.810839 15609 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:39.811153 15609 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:39.819025 15609 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:39.819193 15609 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:39.819506 15609 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:45:39.831936 15609 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:45644 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:39.840004 15609 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:39.841749 15609 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 168819us I0517 06:45:39.842167 15609 fs_manager.cc:260] Metadata directory not provided I0517 06:45:39.842298 15609 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:39.842506 15609 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:45:39.842622 15609 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:39.850126 15609 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" format_stamp: "Formatted at 2018-05-17 06:45:39 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:39.850791 15609 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" format_stamp: "Formatted at 2018-05-17 06:45:39 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:39.859804 15609 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:45:39.860886 15609 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:39.861264 15609 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:39.867535 15609 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:45:39.867642 15609 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" format_stamp: "Formatted at 2018-05-17 06:45:39 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:39.867825 15609 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:40.031255 15609 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:40.033620 15609 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:40.034555 15609 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:40.119798 15609 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:37967 I0517 06:45:40.120690 15609 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:45:40.120796 15609 webserver.cc:184] Document root disabled I0517 06:45:40.121995 15609 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:33564/ I0517 06:45:40.122728 15609 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:45:40.125882 15609 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:40.132557 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 15609 I0517 06:45:40.133484 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.4.96.5:0 --local_ip_for_outbound_sockets=127.4.96.5 --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:45644 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:40.143697 15730 heartbeater.cc:346] Connected to a master server at 127.4.96.62:45644 I0517 06:45:40.143939 15730 heartbeater.cc:426] Registering TS with master... I0517 06:45:40.144373 15730 heartbeater.cc:475] Master 127.4.96.62:45644 requested a full tablet report, sending... I0517 06:45:40.145802 15191 ts_manager.cc:81] Registered new tserver with Master: 91e449d11fed4a27bfcb77c3f16b7e1d (127.4.96.4:37967) I0517 06:45:40.146972 15191 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:53063 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:40.236896 15733 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:40.237237 15733 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:40.245157 15733 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:40.245333 15733 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:40.245627 15733 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.5 I0517 06:45:40.257807 15733 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:45644 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.5 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:40.265696 15733 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:40.267446 15733 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 169319us I0517 06:45:40.267850 15733 fs_manager.cc:260] Metadata directory not provided I0517 06:45:40.267988 15733 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:40.268179 15733 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0517 06:45:40.268290 15733 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:40.275869 15733 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "202f595dac51454bbe6a439465ae7aa5" format_stamp: "Formatted at 2018-05-17 06:45:40 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:40.276511 15733 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "202f595dac51454bbe6a439465ae7aa5" format_stamp: "Formatted at 2018-05-17 06:45:40 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:40.286017 15733 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:45:40.287081 15733 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:40.287472 15733 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:40.293660 15733 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.004s I0517 06:45:40.293773 15733 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal uuid: "202f595dac51454bbe6a439465ae7aa5" format_stamp: "Formatted at 2018-05-17 06:45:40 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:40.293983 15733 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:40.302472 15481 heartbeater.cc:467] Master 127.4.96.62:45644 was elected leader, sending a full tablet report... I0517 06:45:40.486768 15733 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:40.489060 15733 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:40.489995 15733 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:40.574853 15733 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.5:38383 I0517 06:45:40.575744 15733 webserver.cc:173] Starting webserver on 127.4.96.5:0 I0517 06:45:40.575850 15733 webserver.cc:184] Document root disabled I0517 06:45:40.577030 15733 webserver.cc:311] Webserver started. Bound to: http://127.4.96.5:41148/ I0517 06:45:40.577749 15733 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb I0517 06:45:40.579149 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 15733 I0517 06:45:40.581151 15733 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:40.605013 15854 heartbeater.cc:346] Connected to a master server at 127.4.96.62:45644 I0517 06:45:40.605295 15854 heartbeater.cc:426] Registering TS with master... I0517 06:45:40.605763 15854 heartbeater.cc:475] Master 127.4.96.62:45644 requested a full tablet report, sending... I0517 06:45:40.607074 15191 ts_manager.cc:81] Registered new tserver with Master: 202f595dac51454bbe6a439465ae7aa5 (127.4.96.5:38383) I0517 06:45:40.608256 15191 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.5:37415 I0517 06:45:40.608786 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:45:40.627245 15191 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46454: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:45:40.663549 15418 tablet_service.cc:759] Processing CreateTablet for tablet 9594dcd0519943b4b65ccfad32bcb045 (table=TestTable [id=80d6775e5044485b9553db72ac200890]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:40.664264 15418 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9594dcd0519943b4b65ccfad32bcb045. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:40.667937 15667 tablet_service.cc:759] Processing CreateTablet for tablet 9594dcd0519943b4b65ccfad32bcb045 (table=TestTable [id=80d6775e5044485b9553db72ac200890]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:40.668767 15667 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9594dcd0519943b4b65ccfad32bcb045. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:40.672989 15293 tablet_service.cc:759] Processing CreateTablet for tablet 9594dcd0519943b4b65ccfad32bcb045 (table=TestTable [id=80d6775e5044485b9553db72ac200890]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:40.674834 15418 ts_tablet_manager.cc:1173] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:40.673843 15293 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9594dcd0519943b4b65ccfad32bcb045. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:40.684655 15667 ts_tablet_manager.cc:1173] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:40.686435 15875 ts_tablet_manager.cc:1028] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d: Bootstrapping tablet I0517 06:45:40.686424 15874 ts_tablet_manager.cc:1028] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: Bootstrapping tablet I0517 06:45:40.686790 15875 tablet_bootstrap.cc:436] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d: Bootstrap starting. I0517 06:45:40.686854 15874 tablet_bootstrap.cc:436] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: Bootstrap starting. I0517 06:45:40.688899 15875 tablet_bootstrap.cc:587] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d: No blocks or log segments found. Creating new log. I0517 06:45:40.690105 15875 log.cc:520] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d: Log is configured to *not* fsync() on all Append() calls I0517 06:45:40.691412 15874 tablet_bootstrap.cc:587] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: No blocks or log segments found. Creating new log. I0517 06:45:40.692896 15874 log.cc:520] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: Log is configured to *not* fsync() on all Append() calls I0517 06:45:40.695855 15293 ts_tablet_manager.cc:1173] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:40.695977 15875 tablet_bootstrap.cc:436] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d: No bootstrap required, opened a new log I0517 06:45:40.696261 15875 ts_tablet_manager.cc:1045] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s I0517 06:45:40.696591 15874 tablet_bootstrap.cc:436] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: No bootstrap required, opened a new log I0517 06:45:40.696941 15874 ts_tablet_manager.cc:1045] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s I0517 06:45:40.705850 15875 raft_consensus.cc:303] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } I0517 06:45:40.706449 15875 raft_consensus.cc:329] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:40.706620 15875 raft_consensus.cc:604] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e449d11fed4a27bfcb77c3f16b7e1d, State: Initialized, Role: FOLLOWER I0517 06:45:40.707919 15878 ts_tablet_manager.cc:1028] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: Bootstrapping tablet I0517 06:45:40.708288 15878 tablet_bootstrap.cc:436] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: Bootstrap starting. I0517 06:45:40.710425 15878 tablet_bootstrap.cc:587] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: No blocks or log segments found. Creating new log. I0517 06:45:40.711563 15878 log.cc:520] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: Log is configured to *not* fsync() on all Append() calls I0517 06:45:40.711347 15874 raft_consensus.cc:303] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } I0517 06:45:40.711874 15874 raft_consensus.cc:329] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:40.712008 15874 raft_consensus.cc:604] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f3c3b664af3423ea5b117fdb6529c92, State: Initialized, Role: FOLLOWER I0517 06:45:40.712378 15874 consensus_queue.cc:227] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } I0517 06:45:40.713950 15875 consensus_queue.cc:227] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } I0517 06:45:40.715255 15874 ts_tablet_manager.cc:1073] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: Time spent starting tablet: real 0.018s user 0.016s sys 0.004s I0517 06:45:40.715692 15879 raft_consensus.cc:435] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:40.716115 15879 raft_consensus.cc:2704] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d: Snoozing failure detection for 2.130s (starting election) I0517 06:45:40.716598 15878 tablet_bootstrap.cc:436] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: No bootstrap required, opened a new log I0517 06:45:40.716970 15878 ts_tablet_manager.cc:1045] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s I0517 06:45:40.722299 15879 raft_consensus.cc:457] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } I0517 06:45:40.724867 15879 leader_election.cc:243] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6af7d63e13c84ed6b72692f9c995491f (127.4.96.1:36956) I0517 06:45:40.725456 15879 leader_election.cc:243] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2:40737) I0517 06:45:40.727847 15606 heartbeater.cc:467] Master 127.4.96.62:45644 was elected leader, sending a full tablet report... I0517 06:45:40.731319 15878 raft_consensus.cc:303] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } I0517 06:45:40.731847 15878 raft_consensus.cc:329] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:40.731995 15878 raft_consensus.cc:604] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af7d63e13c84ed6b72692f9c995491f, State: Initialized, Role: FOLLOWER I0517 06:45:40.732405 15878 consensus_queue.cc:227] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } I0517 06:45:40.743558 15878 ts_tablet_manager.cc:1073] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: Time spent starting tablet: real 0.026s user 0.012s sys 0.012s I0517 06:45:40.779008 15885 raft_consensus.cc:435] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:40.779206 15885 raft_consensus.cc:2704] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: Snoozing failure detection for 1.582s (starting election) I0517 06:45:40.779407 15885 raft_consensus.cc:457] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } I0517 06:45:40.780277 15885 leader_election.cc:243] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2:40737) I0517 06:45:40.780678 15885 leader_election.cc:243] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 91e449d11fed4a27bfcb77c3f16b7e1d (127.4.96.4:37967) W0517 06:45:40.783051 15482 tablet.cc:1678] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:40.716152 15730 heartbeater.cc:467] Master 127.4.96.62:45644 was elected leader, sending a full tablet report... I0517 06:45:40.805685 15438 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9594dcd0519943b4b65ccfad32bcb045" candidate_uuid: "6af7d63e13c84ed6b72692f9c995491f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3c3b664af3423ea5b117fdb6529c92" is_pre_election: true I0517 06:45:40.806128 15438 raft_consensus.cc:2704] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: Snoozing failure detection for 2.007s (vote granted) I0517 06:45:40.806299 15438 raft_consensus.cc:2219] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af7d63e13c84ed6b72692f9c995491f in term 0. I0517 06:45:40.807202 15245 leader_election.cc:379] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [CANDIDATE]: Term 1 pre-election: Vote granted by peer 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2:40737) I0517 06:45:40.807384 15245 leader_election.cc:270] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:45:40.807813 15885 raft_consensus.cc:2704] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: Snoozing failure detection for 2.000s (election complete) I0517 06:45:40.807996 15885 raft_consensus.cc:2511] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:45:40.808147 15885 raft_consensus.cc:435] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:45:40.808300 15885 raft_consensus.cc:2704] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: Snoozing failure detection for 2.131s (starting election) I0517 06:45:40.808473 15885 raft_consensus.cc:2756] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:40.811256 15885 raft_consensus.cc:457] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } I0517 06:45:40.811900 15885 leader_election.cc:243] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [CANDIDATE]: Term 1 election: Requesting vote from peer 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2:40737) I0517 06:45:40.812191 15885 leader_election.cc:243] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [CANDIDATE]: Term 1 election: Requesting vote from peer 91e449d11fed4a27bfcb77c3f16b7e1d (127.4.96.4:37967) I0517 06:45:40.812964 15438 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9594dcd0519943b4b65ccfad32bcb045" candidate_uuid: "6af7d63e13c84ed6b72692f9c995491f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3c3b664af3423ea5b117fdb6529c92" I0517 06:45:40.813287 15438 raft_consensus.cc:2756] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:40.813529 15438 raft_consensus.cc:2704] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: Snoozing failure detection for 2.109s (vote granted) I0517 06:45:40.722751 15875 ts_tablet_manager.cc:1073] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d: Time spent starting tablet: real 0.026s user 0.012s sys 0.008s I0517 06:45:40.816190 15438 raft_consensus.cc:2219] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af7d63e13c84ed6b72692f9c995491f in term 1. I0517 06:45:40.816984 15245 leader_election.cc:379] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [CANDIDATE]: Term 1 election: Vote granted by peer 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2:40737) I0517 06:45:40.817191 15245 leader_election.cc:270] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:40.817621 15885 raft_consensus.cc:2704] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: Snoozing failure detection for 2.043s (election complete) I0517 06:45:40.817821 15885 raft_consensus.cc:2511] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:40.827154 15885 raft_consensus.cc:571] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 1 LEADER]: Becoming Leader. State: Replica: 6af7d63e13c84ed6b72692f9c995491f, State: Running, Role: LEADER I0517 06:45:40.827657 15885 consensus_queue.cc:204] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } I0517 06:45:40.833873 15687 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9594dcd0519943b4b65ccfad32bcb045" candidate_uuid: "6af7d63e13c84ed6b72692f9c995491f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" is_pre_election: true I0517 06:45:40.834399 15687 raft_consensus.cc:2704] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d: Snoozing failure detection for 2.186s (vote granted) I0517 06:45:40.834620 15687 raft_consensus.cc:2219] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af7d63e13c84ed6b72692f9c995491f in term 0. I0517 06:45:40.835892 15245 leader_election.cc:379] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [CANDIDATE]: Term 1 pre-election: Vote granted by peer 91e449d11fed4a27bfcb77c3f16b7e1d (127.4.96.4:37967) I0517 06:45:40.836509 15687 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9594dcd0519943b4b65ccfad32bcb045" candidate_uuid: "6af7d63e13c84ed6b72692f9c995491f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" I0517 06:45:40.836881 15687 raft_consensus.cc:2756] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:40.837127 15687 raft_consensus.cc:2704] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d: Snoozing failure detection for 2.004s (vote granted) I0517 06:45:40.839975 15687 raft_consensus.cc:2219] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af7d63e13c84ed6b72692f9c995491f in term 1. I0517 06:45:40.840739 15245 leader_election.cc:379] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [CANDIDATE]: Term 1 election: Vote granted by peer 91e449d11fed4a27bfcb77c3f16b7e1d (127.4.96.4:37967) I0517 06:45:40.841179 15438 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9594dcd0519943b4b65ccfad32bcb045" candidate_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3c3b664af3423ea5b117fdb6529c92" is_pre_election: true I0517 06:45:40.841572 15438 raft_consensus.cc:2149] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 91e449d11fed4a27bfcb77c3f16b7e1d in current term 1: Already voted for candidate 6af7d63e13c84ed6b72692f9c995491f in this term. I0517 06:45:40.842566 15619 leader_election.cc:393] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [CANDIDATE]: Term 1 pre-election: Vote denied by peer 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2:40737). Message: Invalid argument: T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 91e449d11fed4a27bfcb77c3f16b7e1d in current term 1: Already voted for candidate 6af7d63e13c84ed6b72692f9c995491f in this term. I0517 06:45:40.839619 15313 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9594dcd0519943b4b65ccfad32bcb045" candidate_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af7d63e13c84ed6b72692f9c995491f" is_pre_election: true I0517 06:45:40.873117 15313 raft_consensus.cc:2180] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 91e449d11fed4a27bfcb77c3f16b7e1d for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:45:40.874258 15618 leader_election.cc:393] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [CANDIDATE]: Term 1 pre-election: Vote denied by peer 6af7d63e13c84ed6b72692f9c995491f (127.4.96.1:36956). Message: Invalid argument: T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 91e449d11fed4a27bfcb77c3f16b7e1d for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:45:40.874472 15618 leader_election.cc:270] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0517 06:45:40.874934 15882 raft_consensus.cc:2704] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d: Snoozing failure detection for 2.117s (election complete) I0517 06:45:40.874603 15191 catalog_manager.cc:3686] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f reported cstate change: term changed from 0 to 1, leader changed from to 6af7d63e13c84ed6b72692f9c995491f (127.4.96.1). New cstate: current_term: 1 leader_uuid: "6af7d63e13c84ed6b72692f9c995491f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } } I0517 06:45:40.875171 15882 raft_consensus.cc:2459] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W0517 06:45:40.876696 15731 tablet.cc:1678] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:40.905897 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:45:40.908033 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6af7d63e13c84ed6b72692f9c995491f to finish bootstrapping I0517 06:45:40.917456 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4f3c3b664af3423ea5b117fdb6529c92 to finish bootstrapping I0517 06:45:40.926028 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5ce72ea2903c48f792f6c4f583dbeb50 to finish bootstrapping I0517 06:45:40.936157 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 91e449d11fed4a27bfcb77c3f16b7e1d to finish bootstrapping I0517 06:45:40.945088 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 202f595dac51454bbe6a439465ae7aa5 to finish bootstrapping I0517 06:45:40.955108 4480 test_util.cc:200] Using random seed: 789774292 W0517 06:45:40.969681 15357 tablet.cc:1678] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:40.970474 4480 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0517 06:45:41.021951 15687 raft_consensus.cc:1057] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 1 FOLLOWER]: Refusing update from remote peer 6af7d63e13c84ed6b72692f9c995491f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0517 06:45:41.022116 15438 raft_consensus.cc:1057] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [term 1 FOLLOWER]: Refusing update from remote peer 6af7d63e13c84ed6b72692f9c995491f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0517 06:45:41.023175 15889 consensus_queue.cc:963] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:41.023771 15885 consensus_queue.cc:963] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0517 06:45:41.140033 15899 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:45:41.611208 15854 heartbeater.cc:467] Master 127.4.96.62:45644 was elected leader, sending a full tablet report... I0517 06:45:41.673995 15313 consensus_peers.cc:456] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f -> Peer 91e449d11fed4a27bfcb77c3f16b7e1d (127.4.96.4:37967): Closing peer: 91e449d11fed4a27bfcb77c3f16b7e1d I0517 06:45:41.674237 15313 consensus_peers.cc:456] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f -> Peer 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2:40737): Closing peer: 4f3c3b664af3423ea5b117fdb6529c92 I0517 06:45:41.674599 15313 consensus_queue.cc:204] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 1.59, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } peers { permanent_uuid: "202f595dac51454bbe6a439465ae7aa5" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 38383 } } I0517 06:45:41.679514 15686 raft_consensus.cc:1057] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 1 FOLLOWER]: Refusing update from remote peer 6af7d63e13c84ed6b72692f9c995491f: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch) I0517 06:45:41.679824 15437 raft_consensus.cc:1057] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [term 1 FOLLOWER]: Refusing update from remote peer 6af7d63e13c84ed6b72692f9c995491f: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch) I0517 06:45:41.680718 15889 consensus_queue.cc:963] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 58, Time since last communication: 0.000s I0517 06:45:41.681205 15910 consensus_queue.cc:963] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s I0517 06:45:41.686053 15910 raft_consensus.cc:2658] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 1 LEADER]: Committing config change with OpId 1.60: config changed from index -1 to 60, VOTER 202f595dac51454bbe6a439465ae7aa5 (127.4.96.5) added. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } peers { permanent_uuid: "202f595dac51454bbe6a439465ae7aa5" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 38383 } } } I0517 06:45:41.687624 15437 raft_consensus.cc:2658] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index -1 to 60, VOTER 202f595dac51454bbe6a439465ae7aa5 (127.4.96.5) added. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } peers { permanent_uuid: "202f595dac51454bbe6a439465ae7aa5" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 38383 } } } W0517 06:45:41.692245 15247 consensus_peers.cc:434] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f -> Peer 202f595dac51454bbe6a439465ae7aa5 (127.4.96.5:38383): Couldn't send request to peer 202f595dac51454bbe6a439465ae7aa5 for tablet 9594dcd0519943b4b65ccfad32bcb045. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9594dcd0519943b4b65ccfad32bcb045. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:45:41.695534 15191 catalog_manager.cc:3686] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f reported cstate change: config changed from index -1 to 60, VOTER 202f595dac51454bbe6a439465ae7aa5 (127.4.96.5) added. New cstate: current_term: 1 leader_uuid: "6af7d63e13c84ed6b72692f9c995491f" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } peers { permanent_uuid: "202f595dac51454bbe6a439465ae7aa5" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 38383 } } } I0517 06:45:41.699287 15687 raft_consensus.cc:2658] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index -1 to 60, VOTER 202f595dac51454bbe6a439465ae7aa5 (127.4.96.5) added. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } peers { permanent_uuid: "202f595dac51454bbe6a439465ae7aa5" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 38383 } } } I0517 06:45:42.178771 15990 ts_tablet_manager.cc:669] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5: Initiating tablet copy from peer 6af7d63e13c84ed6b72692f9c995491f (127.4.96.1:36956) I0517 06:45:42.179203 15990 tablet_copy_client.cc:239] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.1:36956 I0517 06:45:42.193060 15333 tablet_copy_service.cc:133] P 6af7d63e13c84ed6b72692f9c995491f: Received BeginTabletCopySession request for tablet 9594dcd0519943b4b65ccfad32bcb045 from peer 202f595dac51454bbe6a439465ae7aa5 ({username='slave'} at 127.4.96.5:33671) I0517 06:45:42.193377 15333 tablet_copy_service.cc:154] P 6af7d63e13c84ed6b72692f9c995491f: Beginning new tablet copy session on tablet 9594dcd0519943b4b65ccfad32bcb045 from peer 202f595dac51454bbe6a439465ae7aa5 at {username='slave'} at 127.4.96.5:33671: session id = 202f595dac51454bbe6a439465ae7aa5-9594dcd0519943b4b65ccfad32bcb045 I0517 06:45:42.198374 15333 tablet_copy_source_session.cc:202] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:45:42.202090 15990 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9594dcd0519943b4b65ccfad32bcb045. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:42.216089 15990 ts_tablet_manager.cc:1173] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:45:42.216930 15990 tablet_copy_client.cc:565] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5: tablet copy: Starting download of 0 data blocks... I0517 06:45:42.217236 15990 tablet_copy_client.cc:528] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5: tablet copy: Starting download of 1 WAL segments... I0517 06:45:42.220805 15990 tablet_copy_client.cc:404] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:45:42.225155 15990 ts_tablet_manager.cc:1028] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5: Bootstrapping tablet I0517 06:45:42.225500 15990 tablet_bootstrap.cc:436] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5: Bootstrap starting. I0517 06:45:42.254556 15990 log.cc:520] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5: Log is configured to *not* fsync() on all Append() calls I0517 06:45:42.470297 15990 tablet_bootstrap.cc:436] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5: Bootstrap replayed 1/1 log segments. Stats: ops{read=115 overwritten=0 applied=115 ignored=0} inserts{seen=113 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:45:42.471112 15990 tablet_bootstrap.cc:436] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5: Bootstrap complete. I0517 06:45:42.471524 15990 ts_tablet_manager.cc:1045] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5: Time spent bootstrapping tablet: real 0.246s user 0.184s sys 0.016s I0517 06:45:42.480672 15990 raft_consensus.cc:303] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } peers { permanent_uuid: "202f595dac51454bbe6a439465ae7aa5" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 38383 } } I0517 06:45:42.481518 15990 raft_consensus.cc:604] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202f595dac51454bbe6a439465ae7aa5, State: Initialized, Role: FOLLOWER I0517 06:45:42.482208 15990 consensus_queue.cc:227] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 115, Last appended: 1.115, Last appended by leader: 115, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "4f3c3b664af3423ea5b117fdb6529c92" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40737 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } peers { permanent_uuid: "202f595dac51454bbe6a439465ae7aa5" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 38383 } } I0517 06:45:42.495343 15990 ts_tablet_manager.cc:1073] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5: Time spent starting tablet: real 0.024s user 0.008s sys 0.012s I0517 06:45:42.496546 15333 tablet_copy_service.cc:323] P 6af7d63e13c84ed6b72692f9c995491f: Request end of tablet copy session 202f595dac51454bbe6a439465ae7aa5-9594dcd0519943b4b65ccfad32bcb045 received from {username='slave'} at 127.4.96.5:33671 I0517 06:45:42.496778 15333 tablet_copy_service.cc:411] P 6af7d63e13c84ed6b72692f9c995491f: Ending tablet copy session 202f595dac51454bbe6a439465ae7aa5-9594dcd0519943b4b65ccfad32bcb045 on tablet 9594dcd0519943b4b65ccfad32bcb045 with peer 202f595dac51454bbe6a439465ae7aa5 I0517 06:45:42.623739 15811 raft_consensus.cc:1002] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.59->[1.60-1.160] Dedup: 1.115->[1.116-1.160] I0517 06:45:43.095118 15313 consensus_peers.cc:456] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f -> Peer 202f595dac51454bbe6a439465ae7aa5 (127.4.96.5:38383): Closing peer: 202f595dac51454bbe6a439465ae7aa5 I0517 06:45:43.095337 15313 consensus_peers.cc:456] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f -> Peer 91e449d11fed4a27bfcb77c3f16b7e1d (127.4.96.4:37967): Closing peer: 91e449d11fed4a27bfcb77c3f16b7e1d I0517 06:45:43.095500 15313 consensus_peers.cc:456] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f -> Peer 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2:40737): Closing peer: 4f3c3b664af3423ea5b117fdb6529c92 I0517 06:45:43.095880 15313 consensus_queue.cc:204] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 200, Committed index: 200, Last appended: 1.201, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } peers { permanent_uuid: "202f595dac51454bbe6a439465ae7aa5" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 38383 } } I0517 06:45:43.100183 15810 raft_consensus.cc:1057] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5 [term 1 FOLLOWER]: Refusing update from remote peer 6af7d63e13c84ed6b72692f9c995491f: Log matching property violated. Preceding OpId in replica: term: 1 index: 201. Preceding OpId from leader: term: 1 index: 202. (index mismatch) I0517 06:45:43.100392 15686 raft_consensus.cc:1057] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 1 FOLLOWER]: Refusing update from remote peer 6af7d63e13c84ed6b72692f9c995491f: Log matching property violated. Preceding OpId in replica: term: 1 index: 201. Preceding OpId from leader: term: 1 index: 202. (index mismatch) I0517 06:45:43.101449 15910 consensus_queue.cc:963] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 202, Last known committed idx: 200, Time since last communication: 0.000s I0517 06:45:43.101933 15919 consensus_queue.cc:963] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [LEADER]: Connected to new peer: Peer: permanent_uuid: "202f595dac51454bbe6a439465ae7aa5" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 202, Last known committed idx: 200, Time since last communication: 0.000s I0517 06:45:43.106112 15919 raft_consensus.cc:2658] T 9594dcd0519943b4b65ccfad32bcb045 P 6af7d63e13c84ed6b72692f9c995491f [term 1 LEADER]: Committing config change with OpId 1.202: config changed from index 60 to 202, VOTER 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2) evicted. New config: { opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } peers { permanent_uuid: "202f595dac51454bbe6a439465ae7aa5" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 38383 } } } I0517 06:45:43.107368 15686 raft_consensus.cc:2658] T 9594dcd0519943b4b65ccfad32bcb045 P 91e449d11fed4a27bfcb77c3f16b7e1d [term 1 FOLLOWER]: Committing config change with OpId 1.202: config changed from index 60 to 202, VOTER 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2) evicted. New config: { opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } peers { permanent_uuid: "202f595dac51454bbe6a439465ae7aa5" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 38383 } } } I0517 06:45:43.108548 15810 raft_consensus.cc:2658] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5 [term 1 FOLLOWER]: Committing config change with OpId 1.202: config changed from index 60 to 202, VOTER 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2) evicted. New config: { opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } peers { permanent_uuid: "202f595dac51454bbe6a439465ae7aa5" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 38383 } } } I0517 06:45:43.118190 15190 catalog_manager.cc:3686] T 9594dcd0519943b4b65ccfad32bcb045 P 202f595dac51454bbe6a439465ae7aa5 reported cstate change: config changed from index 60 to 202, VOTER 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2) evicted. New cstate: current_term: 1 leader_uuid: "6af7d63e13c84ed6b72692f9c995491f" committed_config { opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "6af7d63e13c84ed6b72692f9c995491f" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 36956 } } peers { permanent_uuid: "91e449d11fed4a27bfcb77c3f16b7e1d" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 37967 } } peers { permanent_uuid: "202f595dac51454bbe6a439465ae7aa5" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 38383 } } } I0517 06:45:43.151595 15190 catalog_manager.cc:3013] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 9594dcd0519943b4b65ccfad32bcb045 on 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2:40737) (TS 4f3c3b664af3423ea5b117fdb6529c92 not found in new config with opid_index 202) I0517 06:45:43.153084 15418 tablet_service.cc:800] Processing DeleteTablet for tablet 9594dcd0519943b4b65ccfad32bcb045 with delete_type TABLET_DATA_TOMBSTONED (TS 4f3c3b664af3423ea5b117fdb6529c92 not found in new config with opid_index 202) from {username='slave'} at 127.0.0.1:60477 I0517 06:45:43.154676 16050 tablet_replica.cc:261] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: stopping tablet replica I0517 06:45:43.154961 16050 maintenance_manager.cc:237] P 4f3c3b664af3423ea5b117fdb6529c92: Unregistered op CompactRowSetsOp(9594dcd0519943b4b65ccfad32bcb045) I0517 06:45:43.155138 16050 maintenance_manager.cc:237] P 4f3c3b664af3423ea5b117fdb6529c92: Unregistered op MinorDeltaCompactionOp(9594dcd0519943b4b65ccfad32bcb045) I0517 06:45:43.155270 16050 maintenance_manager.cc:237] P 4f3c3b664af3423ea5b117fdb6529c92: Unregistered op MajorDeltaCompactionOp(9594dcd0519943b4b65ccfad32bcb045) I0517 06:45:43.155449 16050 maintenance_manager.cc:237] P 4f3c3b664af3423ea5b117fdb6529c92: Unregistered op UndoDeltaBlockGCOp(9594dcd0519943b4b65ccfad32bcb045) I0517 06:45:43.155660 16050 maintenance_manager.cc:237] P 4f3c3b664af3423ea5b117fdb6529c92: Unregistered op FlushMRSOp(9594dcd0519943b4b65ccfad32bcb045) I0517 06:45:43.155846 16050 maintenance_manager.cc:237] P 4f3c3b664af3423ea5b117fdb6529c92: Unregistered op FlushDeltaMemStoresOp(9594dcd0519943b4b65ccfad32bcb045) I0517 06:45:43.156028 16050 maintenance_manager.cc:237] P 4f3c3b664af3423ea5b117fdb6529c92: Unregistered op LogGCOp(9594dcd0519943b4b65ccfad32bcb045) I0517 06:45:43.156208 16050 raft_consensus.cc:2016] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [term 1 FOLLOWER]: Raft consensus shutting down. I0517 06:45:43.156479 16050 pending_rounds.cc:63] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: Trying to abort 1 pending transactions. I0517 06:45:43.156630 16050 pending_rounds.cc:68] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: Aborting transaction as it isn't in flight: id { term: 1 index: 201 } timestamp: 6252705968495235072 op_type: WRITE_OP write_request { tablet_id: "9594dcd0519943b4b65ccfad32bcb045" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\355\241\367A\275~\232i\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "1d8518589a02d2bf" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6252705968442949632 } request_id { client_id: "d92edd78d00b4991b439e4d4c6595cad" seq_no: 198 first_incomplete_seq_no: 198 attempt_no: 0 } I0517 06:45:43.158404 16050 raft_consensus.cc:2043] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92 [term 1 FOLLOWER]: Raft consensus is shut down! I0517 06:45:43.161137 16050 ts_tablet_manager.cc:1369] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0517 06:45:43.169885 16050 ts_tablet_manager.cc:1382] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.201 I0517 06:45:43.170162 16050 log.cc:981] T 9594dcd0519943b4b65ccfad32bcb045 P 4f3c3b664af3423ea5b117fdb6529c92: Deleting WAL directory at /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/wals/9594dcd0519943b4b65ccfad32bcb045 I0517 06:45:43.171551 15173 catalog_manager.cc:2991] TS 4f3c3b664af3423ea5b117fdb6529c92 (127.4.96.2:40737): tablet 9594dcd0519943b4b65ccfad32bcb045 (table TestTable [id=80d6775e5044485b9553db72ac200890]) successfully deleted W0517 06:45:44.145483 15248 debug-util.cc:397] Leaking SignalData structure 0x7b080002b060 after lost signal to thread 15234 W0517 06:45:44.146350 15248 debug-util.cc:397] Leaking SignalData structure 0x7b080009c6e0 after lost signal to thread 15354 Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- fc6df897b46e454f8b2e219dc5f39d44 | 127.4.96.62:45644 | HEALTHY Tablet Server Summary UUID | Address | Status ----------------------------------+------------------+--------- 202f595dac51454bbe6a439465ae7aa5 | 127.4.96.5:38383 | HEALTHY 4f3c3b664af3423ea5b117fdb6529c92 | 127.4.96.2:40737 | HEALTHY 5ce72ea2903c48f792f6c4f583dbeb50 | 127.4.96.3:49834 | HEALTHY 6af7d63e13c84ed6b72692f9c995491f | 127.4.96.1:36956 | HEALTHY 91e449d11fed4a27bfcb77c3f16b7e1d | 127.4.96.4:37967 | HEALTHY Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 5 Tables | 1 Tablets | 1 Replicas | 3 OK I0517 06:45:44.320907 4480 log_verifier.cc:121] Checking tablet 9594dcd0519943b4b65ccfad32bcb045 I0517 06:45:44.693751 4480 log_verifier.cc:172] Verified matching terms for 339 ops in tablet 9594dcd0519943b4b65ccfad32bcb045 I0517 06:45:44.699604 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 15225 W0517 06:45:44.731864 15172 connection.cc:512] client connection to 127.4.96.1:36956 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:36956): Connection reset by peer (error 104) W0517 06:45:44.731921 15618 connection.cc:512] client connection to 127.4.96.1:36956 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:36956): Connection reset by peer (error 104) W0517 06:45:44.732729 15172 connection.cc:512] server connection from 127.4.96.1:54550 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:54550): Connection reset by peer (error 104) W0517 06:45:44.732880 15745 connection.cc:512] server connection from 127.4.96.1:42471 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:42471): Connection reset by peer (error 104) W0517 06:45:44.733636 15742 connection.cc:512] client connection to 127.4.96.1:36956 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:36956): Connection reset by peer (error 104) W0517 06:45:44.734310 15370 connection.cc:512] server connection from 127.4.96.1:50728 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:50728): Connection reset by peer (error 104) I0517 06:45:44.734907 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 15360 W0517 06:45:44.735442 15162 connection.cc:512] client connection to 127.4.96.1:36956 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:36956): Connection reset by peer (error 104) W0517 06:45:44.736346 15619 connection.cc:512] server connection from 127.4.96.1:44519 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:44519): Connection reset by peer (error 104) W0517 06:45:44.760468 15173 connection.cc:512] client connection to 127.4.96.2:40737 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:40737): Connection reset by peer (error 104) W0517 06:45:44.760512 15172 connection.cc:512] server connection from 127.4.96.2:51460 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:51460): Connection reset by peer (error 104) W0517 06:45:44.760920 15619 connection.cc:512] client connection to 127.4.96.2:40737 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:40737): Connection reset by peer (error 104) W0517 06:45:44.761665 15162 connection.cc:512] client connection to 127.4.96.2:40737 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:40737): Connection reset by peer (error 104) I0517 06:45:44.769263 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 15485 W0517 06:45:44.786281 15162 connection.cc:512] client connection to 127.4.96.3:49834 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:49834): Connection reset by peer (error 104) W0517 06:45:44.786281 15173 connection.cc:512] server connection from 127.4.96.3:35771 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:35771): Connection reset by peer (error 104) I0517 06:45:44.787088 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 15609 W0517 06:45:44.819099 15173 connection.cc:512] client connection to 127.4.96.4:37967 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:37967): Connection reset by peer (error 104) I0517 06:45:44.819238 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 15733 W0517 06:45:44.820075 15162 connection.cc:512] client connection to 127.4.96.4:37967 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:37967): Connection reset by peer (error 104) W0517 06:45:44.826093 15171 connection.cc:512] server connection from 127.4.96.4:53063 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:53063): Connection reset by peer (error 104) W0517 06:45:44.850471 15162 connection.cc:512] client connection to 127.4.96.5:38383 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:38383): Connection reset by peer (error 104) W0517 06:45:44.850491 15171 connection.cc:512] server connection from 127.4.96.5:37415 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:37415): Connection reset by peer (error 104) I0517 06:45:44.851317 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 15163 W0517 06:45:44.854153 15161 connection.cc:512] client connection to 127.4.96.5:38383 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:38383): Connection reset by peer (error 104) W0517 06:45:44.877189 15162 connection.cc:512] client connection to 127.4.96.62:45644 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:45644): Connection reset by peer (error 104) [ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/0 (7352 ms) [ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/1 I0517 06:45:44.897002 4480 test_util.cc:200] Using random seed: 793716182 I0517 06:45:44.899937 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:45:44.900030 4480 ts_itest-base.cc:114] -------------- I0517 06:45:44.900099 4480 ts_itest-base.cc:115] 5 tablet servers I0517 06:45:44.900167 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:45:44.900235 4480 ts_itest-base.cc:117] -------------- I0517 06:45:44.902181 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:36173 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:45.015141 16086 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:45.015450 16086 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:45.024341 16086 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:45.024494 16086 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:45:45.024592 16086 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:45.024684 16086 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:45:45.024794 16086 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:45:45.038615 16086 master_main.cc:73] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:36173 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:45.046231 16086 master_main.cc:80] Initializing master server... I0517 06:45:45.047719 16086 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 171819us I0517 06:45:45.048449 16086 fs_manager.cc:260] Metadata directory not provided I0517 06:45:45.048565 16086 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:45.048744 16086 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:45:45.048857 16086 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:45.056004 16086 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "836396dec77e4248a5273ef503dd0344" format_stamp: "Formatted at 2018-05-17 06:45:45 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:45.056665 16086 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "836396dec77e4248a5273ef503dd0344" format_stamp: "Formatted at 2018-05-17 06:45:45 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:45.065831 16086 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:45:45.066934 16086 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:45.067312 16086 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:45.073629 16086 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:45:45.073735 16086 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "836396dec77e4248a5273ef503dd0344" format_stamp: "Formatted at 2018-05-17 06:45:45 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:45.073940 16086 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:45.236604 16086 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:45.238298 16086 master_main.cc:83] Starting Master server... I0517 06:45:45.277709 16086 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:36173 I0517 06:45:45.278542 16086 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:45:45.278642 16086 webserver.cc:184] Document root disabled I0517 06:45:45.279817 16086 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:44537/ I0517 06:45:45.280541 16086 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:45:45.283040 16147 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:45.285517 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 16086 I0517 06:45:45.286461 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36173 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:45.310884 16147 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344: Bootstrap starting. I0517 06:45:45.313138 16147 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344: No blocks or log segments found. Creating new log. I0517 06:45:45.314321 16147 log.cc:520] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344: Log is configured to *not* fsync() on all Append() calls I0517 06:45:45.322981 16147 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344: No bootstrap required, opened a new log I0517 06:45:45.335206 16147 raft_consensus.cc:303] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836396dec77e4248a5273ef503dd0344" member_type: VOTER } I0517 06:45:45.335528 16147 raft_consensus.cc:329] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:45.335677 16147 raft_consensus.cc:604] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836396dec77e4248a5273ef503dd0344, State: Initialized, Role: FOLLOWER I0517 06:45:45.336134 16147 consensus_queue.cc:227] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836396dec77e4248a5273ef503dd0344" member_type: VOTER } I0517 06:45:45.336553 16147 raft_consensus.cc:343] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:45:45.336735 16147 raft_consensus.cc:435] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:45:45.336931 16147 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344: Snoozing failure detection for 2.185s (starting election) I0517 06:45:45.337117 16147 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:45.339692 16147 raft_consensus.cc:457] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836396dec77e4248a5273ef503dd0344" member_type: VOTER } I0517 06:45:45.340132 16147 leader_election.cc:270] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:45.350482 16150 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344: Snoozing failure detection for 1.877s (election complete) I0517 06:45:45.350767 16150 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:45.370244 16150 raft_consensus.cc:571] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [term 1 LEADER]: Becoming Leader. State: Replica: 836396dec77e4248a5273ef503dd0344, State: Running, Role: LEADER I0517 06:45:45.370584 16150 consensus_queue.cc:204] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836396dec77e4248a5273ef503dd0344" member_type: VOTER } I0517 06:45:45.383918 16151 sys_catalog.cc:335] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "836396dec77e4248a5273ef503dd0344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836396dec77e4248a5273ef503dd0344" member_type: VOTER } } I0517 06:45:45.384505 16151 sys_catalog.cc:338] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [sys.catalog]: This master's current role is: LEADER WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:45.405014 16148 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:45.405339 16148 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:45.413126 16148 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:45.413297 16148 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:45.413604 16148 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:45:45.429404 16148 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36173 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:45.437528 16148 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:45.439399 16148 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 171819us I0517 06:45:45.439843 16148 fs_manager.cc:260] Metadata directory not provided I0517 06:45:45.439970 16148 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:45.440176 16148 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:45:45.440294 16148 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:45.447851 16148 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "c6e6b14e43c24c699915460b0dcc2f61" format_stamp: "Formatted at 2018-05-17 06:45:45 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:45.448529 16148 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "c6e6b14e43c24c699915460b0dcc2f61" format_stamp: "Formatted at 2018-05-17 06:45:45 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:45.383924 16152 sys_catalog.cc:335] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 836396dec77e4248a5273ef503dd0344. Latest consensus state: current_term: 1 leader_uuid: "836396dec77e4248a5273ef503dd0344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836396dec77e4248a5273ef503dd0344" member_type: VOTER } } I0517 06:45:45.466796 16152 sys_catalog.cc:338] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [sys.catalog]: This master's current role is: LEADER I0517 06:45:45.468384 16148 fs_manager.cc:495] Time spent creating directory manager: real 0.020s user 0.012s sys 0.008s I0517 06:45:45.469918 16148 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:45.470422 16148 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:45.385946 16147 sys_catalog.cc:335] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "836396dec77e4248a5273ef503dd0344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836396dec77e4248a5273ef503dd0344" member_type: VOTER } } I0517 06:45:45.470674 16147 sys_catalog.cc:338] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [sys.catalog]: This master's current role is: LEADER I0517 06:45:45.471799 16147 sys_catalog.cc:422] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:45:45.472059 16154 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:45:45.473704 16086 master_main.cc:86] Master server successfully started. I0517 06:45:45.481591 16148 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s I0517 06:45:45.481742 16148 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "c6e6b14e43c24c699915460b0dcc2f61" format_stamp: "Formatted at 2018-05-17 06:45:45 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:45.482043 16148 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:45.482515 16154 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... W0517 06:45:45.489859 16103 tablet.cc:1678] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:45.515986 16154 catalog_manager.cc:899] Generated new certificate authority record I0517 06:45:45.518054 16154 catalog_manager.cc:1021] Loading token signing keys... I0517 06:45:45.526093 16154 catalog_manager.cc:3962] T 00000000000000000000000000000000 P 836396dec77e4248a5273ef503dd0344: Generated new TSK 0 I0517 06:45:45.666085 16148 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:45.668679 16148 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:45.669842 16148 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:45.772825 16148 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:38880 I0517 06:45:45.773730 16148 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:45:45.773835 16148 webserver.cc:184] Document root disabled I0517 06:45:45.775244 16148 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:45605/ I0517 06:45:45.775928 16148 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:45:45.778378 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 16148 I0517 06:45:45.779655 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36173 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:45.786470 16148 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:45.801880 16279 heartbeater.cc:346] Connected to a master server at 127.4.96.62:36173 I0517 06:45:45.802150 16279 heartbeater.cc:426] Registering TS with master... I0517 06:45:45.802594 16279 heartbeater.cc:475] Master 127.4.96.62:36173 requested a full tablet report, sending... I0517 06:45:45.804177 16114 ts_manager.cc:81] Registered new tserver with Master: c6e6b14e43c24c699915460b0dcc2f61 (127.4.96.1:38880) I0517 06:45:45.805836 16114 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:42398 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:45.941678 16283 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:45.942144 16283 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:45.956418 16283 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:45.956645 16283 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:45.957132 16283 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:45:45.977846 16283 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36173 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:45.991129 16283 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:45.993535 16283 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 171819us I0517 06:45:45.994148 16283 fs_manager.cc:260] Metadata directory not provided I0517 06:45:45.994313 16283 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:45.994583 16283 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:45:45.994742 16283 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:46.007632 16283 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "88e96f529a8d442eacbf6b571e51c10c" format_stamp: "Formatted at 2018-05-17 06:45:46 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:46.008568 16283 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "88e96f529a8d442eacbf6b571e51c10c" format_stamp: "Formatted at 2018-05-17 06:45:46 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:46.022851 16283 fs_manager.cc:495] Time spent creating directory manager: real 0.014s user 0.012s sys 0.004s I0517 06:45:46.024508 16283 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:46.025048 16283 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:46.047166 16283 fs_manager.cc:417] Time spent opening block manager: real 0.013s user 0.004s sys 0.012s I0517 06:45:46.047317 16283 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "88e96f529a8d442eacbf6b571e51c10c" format_stamp: "Formatted at 2018-05-17 06:45:46 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:46.047587 16283 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:46.226066 16283 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:46.229168 16283 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:46.230298 16283 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:46.473358 16283 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:37572 I0517 06:45:46.474270 16283 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:45:46.474401 16283 webserver.cc:184] Document root disabled I0517 06:45:46.475782 16283 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:43521/ I0517 06:45:46.476481 16283 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:45:46.480530 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 16283 I0517 06:45:46.481716 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36173 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:46.486719 16283 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:46.507755 16404 heartbeater.cc:346] Connected to a master server at 127.4.96.62:36173 I0517 06:45:46.508060 16404 heartbeater.cc:426] Registering TS with master... I0517 06:45:46.508611 16404 heartbeater.cc:475] Master 127.4.96.62:36173 requested a full tablet report, sending... I0517 06:45:46.510231 16114 ts_manager.cc:81] Registered new tserver with Master: 88e96f529a8d442eacbf6b571e51c10c (127.4.96.2:37572) I0517 06:45:46.511839 16114 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:33855 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:46.600116 16408 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:46.600507 16408 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:46.608433 16408 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:46.608603 16408 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:46.608907 16408 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:45:46.621897 16408 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36173 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:46.630934 16408 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:46.632699 16408 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 172319us I0517 06:45:46.633131 16408 fs_manager.cc:260] Metadata directory not provided I0517 06:45:46.633257 16408 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:46.633461 16408 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:45:46.633572 16408 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:46.641098 16408 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "564d4149aa01431db4135c45439f1c2a" format_stamp: "Formatted at 2018-05-17 06:45:46 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:46.641750 16408 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "564d4149aa01431db4135c45439f1c2a" format_stamp: "Formatted at 2018-05-17 06:45:46 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:46.651254 16408 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:45:46.652351 16408 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:46.652951 16408 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:46.659608 16408 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0517 06:45:46.659736 16408 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "564d4149aa01431db4135c45439f1c2a" format_stamp: "Formatted at 2018-05-17 06:45:46 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:46.659976 16408 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:46.808197 16279 heartbeater.cc:467] Master 127.4.96.62:36173 was elected leader, sending a full tablet report... I0517 06:45:46.862354 16408 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:46.864984 16408 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:46.866128 16408 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:46.966267 16408 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:47712 I0517 06:45:46.967180 16408 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:45:46.967291 16408 webserver.cc:184] Document root disabled I0517 06:45:46.968478 16408 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:48356/ I0517 06:45:46.969161 16408 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:45:46.972786 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 16408 I0517 06:45:46.972980 16408 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:46.974031 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36173 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:46.990365 16529 heartbeater.cc:346] Connected to a master server at 127.4.96.62:36173 I0517 06:45:46.990591 16529 heartbeater.cc:426] Registering TS with master... I0517 06:45:46.990996 16529 heartbeater.cc:475] Master 127.4.96.62:36173 requested a full tablet report, sending... I0517 06:45:46.992262 16114 ts_manager.cc:81] Registered new tserver with Master: 564d4149aa01431db4135c45439f1c2a (127.4.96.3:47712) I0517 06:45:46.993499 16114 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:36935 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:47.079921 16532 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:47.080261 16532 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:47.088124 16532 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:47.088289 16532 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:47.088591 16532 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:45:47.100911 16532 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36173 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:47.109001 16532 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:47.110827 16532 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 172819us I0517 06:45:47.111279 16532 fs_manager.cc:260] Metadata directory not provided I0517 06:45:47.111402 16532 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:47.111605 16532 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:45:47.111721 16532 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:47.119212 16532 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" format_stamp: "Formatted at 2018-05-17 06:45:47 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:47.119870 16532 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" format_stamp: "Formatted at 2018-05-17 06:45:47 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:47.128834 16532 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:45:47.129956 16532 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:47.130355 16532 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:47.136615 16532 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:45:47.136730 16532 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" format_stamp: "Formatted at 2018-05-17 06:45:47 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:47.136935 16532 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:47.299265 16532 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:47.301872 16532 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:47.302819 16532 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:47.386919 16532 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:44014 I0517 06:45:47.387815 16532 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:45:47.387922 16532 webserver.cc:184] Document root disabled I0517 06:45:47.389128 16532 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:43285/ I0517 06:45:47.389818 16532 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:45:47.392112 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 16532 I0517 06:45:47.392251 16532 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:47.393209 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.4.96.5:0 --local_ip_for_outbound_sockets=127.4.96.5 --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36173 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:47.409945 16653 heartbeater.cc:346] Connected to a master server at 127.4.96.62:36173 I0517 06:45:47.410174 16653 heartbeater.cc:426] Registering TS with master... I0517 06:45:47.410585 16653 heartbeater.cc:475] Master 127.4.96.62:36173 requested a full tablet report, sending... I0517 06:45:47.411895 16114 ts_manager.cc:81] Registered new tserver with Master: b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014) I0517 06:45:47.412971 16114 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:58638 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:47.499825 16656 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:47.500149 16656 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:47.507953 16656 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:47.508121 16656 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:47.508428 16656 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.5 I0517 06:45:47.514966 16404 heartbeater.cc:467] Master 127.4.96.62:36173 was elected leader, sending a full tablet report... I0517 06:45:47.521160 16656 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36173 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.5 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:47.529047 16656 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:47.530773 16656 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 172819us I0517 06:45:47.531181 16656 fs_manager.cc:260] Metadata directory not provided I0517 06:45:47.531302 16656 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:47.531481 16656 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0517 06:45:47.531586 16656 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:47.538877 16656 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "1f2902b4a6b24c379dbe0f755512f4a3" format_stamp: "Formatted at 2018-05-17 06:45:47 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:47.539487 16656 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "1f2902b4a6b24c379dbe0f755512f4a3" format_stamp: "Formatted at 2018-05-17 06:45:47 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:47.548373 16656 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:45:47.549485 16656 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:47.549867 16656 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:47.556085 16656 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:45:47.556188 16656 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal uuid: "1f2902b4a6b24c379dbe0f755512f4a3" format_stamp: "Formatted at 2018-05-17 06:45:47 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:47.556401 16656 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:47.702744 16656 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:47.705032 16656 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:47.705945 16656 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:47.790354 16656 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.5:41271 I0517 06:45:47.791216 16656 webserver.cc:173] Starting webserver on 127.4.96.5:0 I0517 06:45:47.791309 16656 webserver.cc:184] Document root disabled I0517 06:45:47.792476 16656 webserver.cc:311] Webserver started. Bound to: http://127.4.96.5:54427/ I0517 06:45:47.793139 16656 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb I0517 06:45:47.796070 16656 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:47.801441 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 16656 I0517 06:45:47.816378 16777 heartbeater.cc:346] Connected to a master server at 127.4.96.62:36173 I0517 06:45:47.816632 16777 heartbeater.cc:426] Registering TS with master... I0517 06:45:47.817147 16777 heartbeater.cc:475] Master 127.4.96.62:36173 requested a full tablet report, sending... I0517 06:45:47.818374 16114 ts_manager.cc:81] Registered new tserver with Master: 1f2902b4a6b24c379dbe0f755512f4a3 (127.4.96.5:41271) I0517 06:45:47.819808 16114 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.5:48217 I0517 06:45:47.820227 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:45:47.837234 16114 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37797: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:45:47.899351 16341 tablet_service.cc:759] Processing CreateTablet for tablet d9374f3bfb9043bdb65da277ea8671c7 (table=TestTable [id=4e568cd61841461ca88e0c0fab6ce7d2]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:47.900189 16341 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d9374f3bfb9043bdb65da277ea8671c7. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:47.900768 16590 tablet_service.cc:759] Processing CreateTablet for tablet d9374f3bfb9043bdb65da277ea8671c7 (table=TestTable [id=4e568cd61841461ca88e0c0fab6ce7d2]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:47.901098 16466 tablet_service.cc:759] Processing CreateTablet for tablet d9374f3bfb9043bdb65da277ea8671c7 (table=TestTable [id=4e568cd61841461ca88e0c0fab6ce7d2]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:47.901605 16590 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d9374f3bfb9043bdb65da277ea8671c7. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:47.901854 16466 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d9374f3bfb9043bdb65da277ea8671c7. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:47.912536 16341 ts_tablet_manager.cc:1173] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:47.917580 16590 ts_tablet_manager.cc:1173] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:47.917824 16797 ts_tablet_manager.cc:1028] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c: Bootstrapping tablet I0517 06:45:47.918395 16797 tablet_bootstrap.cc:436] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c: Bootstrap starting. I0517 06:45:47.918982 16798 ts_tablet_manager.cc:1028] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5: Bootstrapping tablet I0517 06:45:47.918906 16466 ts_tablet_manager.cc:1173] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:47.919320 16798 tablet_bootstrap.cc:436] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5: Bootstrap starting. I0517 06:45:47.921404 16797 tablet_bootstrap.cc:587] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c: No blocks or log segments found. Creating new log. I0517 06:45:47.921479 16798 tablet_bootstrap.cc:587] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5: No blocks or log segments found. Creating new log. I0517 06:45:47.922979 16798 log.cc:520] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5: Log is configured to *not* fsync() on all Append() calls I0517 06:45:47.922981 16797 log.cc:520] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c: Log is configured to *not* fsync() on all Append() calls I0517 06:45:47.924429 16799 ts_tablet_manager.cc:1028] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: Bootstrapping tablet I0517 06:45:47.925997 16799 tablet_bootstrap.cc:436] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: Bootstrap starting. I0517 06:45:47.926784 16798 tablet_bootstrap.cc:436] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5: No bootstrap required, opened a new log I0517 06:45:47.927067 16798 ts_tablet_manager.cc:1045] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s I0517 06:45:47.927319 16797 tablet_bootstrap.cc:436] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c: No bootstrap required, opened a new log I0517 06:45:47.927664 16797 ts_tablet_manager.cc:1045] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s I0517 06:45:47.928807 16799 tablet_bootstrap.cc:587] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: No blocks or log segments found. Creating new log. I0517 06:45:47.930382 16799 log.cc:520] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: Log is configured to *not* fsync() on all Append() calls I0517 06:45:47.933063 16799 tablet_bootstrap.cc:436] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: No bootstrap required, opened a new log I0517 06:45:47.933411 16799 ts_tablet_manager.cc:1045] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s I0517 06:45:47.938599 16798 raft_consensus.cc:303] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } I0517 06:45:47.939152 16798 raft_consensus.cc:329] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:47.939314 16798 raft_consensus.cc:604] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0513a4bbf7d4f938b5eecba5c0967f5, State: Initialized, Role: FOLLOWER I0517 06:45:47.939749 16798 consensus_queue.cc:227] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } I0517 06:45:47.942615 16797 raft_consensus.cc:303] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } I0517 06:45:47.943375 16797 raft_consensus.cc:329] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:47.943576 16797 raft_consensus.cc:604] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e96f529a8d442eacbf6b571e51c10c, State: Initialized, Role: FOLLOWER I0517 06:45:47.944212 16797 consensus_queue.cc:227] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } I0517 06:45:47.945540 16653 heartbeater.cc:467] Master 127.4.96.62:36173 was elected leader, sending a full tablet report... I0517 06:45:47.948578 16799 raft_consensus.cc:303] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } I0517 06:45:47.949306 16799 raft_consensus.cc:329] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:47.949494 16799 raft_consensus.cc:604] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564d4149aa01431db4135c45439f1c2a, State: Initialized, Role: FOLLOWER I0517 06:45:47.950093 16799 consensus_queue.cc:227] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } I0517 06:45:47.953059 16529 heartbeater.cc:467] Master 127.4.96.62:36173 was elected leader, sending a full tablet report... I0517 06:45:47.953851 16797 ts_tablet_manager.cc:1073] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c: Time spent starting tablet: real 0.026s user 0.020s sys 0.004s I0517 06:45:47.954942 16799 ts_tablet_manager.cc:1073] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: Time spent starting tablet: real 0.021s user 0.020s sys 0.000s I0517 06:45:47.961900 16803 raft_consensus.cc:435] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:47.962121 16803 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5: Snoozing failure detection for 1.531s (starting election) I0517 06:45:47.962321 16803 raft_consensus.cc:457] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } I0517 06:45:47.963163 16803 leader_election.cc:243] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 564d4149aa01431db4135c45439f1c2a (127.4.96.3:47712) I0517 06:45:47.963572 16803 leader_election.cc:243] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 88e96f529a8d442eacbf6b571e51c10c (127.4.96.2:37572) I0517 06:45:47.945940 16798 ts_tablet_manager.cc:1073] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s I0517 06:45:48.051805 16361 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d9374f3bfb9043bdb65da277ea8671c7" candidate_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88e96f529a8d442eacbf6b571e51c10c" is_pre_election: true I0517 06:45:48.052284 16361 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c: Snoozing failure detection for 1.718s (vote granted) I0517 06:45:48.052486 16361 raft_consensus.cc:2219] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0513a4bbf7d4f938b5eecba5c0967f5 in term 0. I0517 06:45:48.053469 16544 leader_election.cc:379] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 88e96f529a8d442eacbf6b571e51c10c (127.4.96.2:37572) I0517 06:45:48.053663 16544 leader_election.cc:270] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:45:48.054090 16803 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5: Snoozing failure detection for 1.623s (election complete) I0517 06:45:48.054282 16803 raft_consensus.cc:2511] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:45:48.054442 16803 raft_consensus.cc:435] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:45:48.054595 16803 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5: Snoozing failure detection for 2.248s (starting election) I0517 06:45:48.054759 16803 raft_consensus.cc:2756] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:48.055289 16486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d9374f3bfb9043bdb65da277ea8671c7" candidate_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564d4149aa01431db4135c45439f1c2a" is_pre_election: true I0517 06:45:48.055867 16486 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: Snoozing failure detection for 1.815s (vote granted) I0517 06:45:48.056118 16486 raft_consensus.cc:2219] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0513a4bbf7d4f938b5eecba5c0967f5 in term 0. I0517 06:45:48.057214 16542 leader_election.cc:379] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 564d4149aa01431db4135c45439f1c2a (127.4.96.3:47712) I0517 06:45:48.057883 16803 raft_consensus.cc:457] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } I0517 06:45:48.058562 16803 leader_election.cc:243] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [CANDIDATE]: Term 1 election: Requesting vote from peer 564d4149aa01431db4135c45439f1c2a (127.4.96.3:47712) I0517 06:45:48.058856 16803 leader_election.cc:243] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [CANDIDATE]: Term 1 election: Requesting vote from peer 88e96f529a8d442eacbf6b571e51c10c (127.4.96.2:37572) I0517 06:45:48.059779 16361 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d9374f3bfb9043bdb65da277ea8671c7" candidate_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88e96f529a8d442eacbf6b571e51c10c" I0517 06:45:48.059697 16486 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d9374f3bfb9043bdb65da277ea8671c7" candidate_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564d4149aa01431db4135c45439f1c2a" I0517 06:45:48.060083 16361 raft_consensus.cc:2756] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:48.060101 16486 raft_consensus.cc:2756] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:48.060286 16361 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c: Snoozing failure detection for 2.226s (vote granted) I0517 06:45:48.060341 16486 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: Snoozing failure detection for 2.067s (vote granted) I0517 06:45:48.062986 16361 raft_consensus.cc:2219] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0513a4bbf7d4f938b5eecba5c0967f5 in term 1. I0517 06:45:48.063637 16544 leader_election.cc:379] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [CANDIDATE]: Term 1 election: Vote granted by peer 88e96f529a8d442eacbf6b571e51c10c (127.4.96.2:37572) I0517 06:45:48.063804 16544 leader_election.cc:270] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:48.064141 16803 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5: Snoozing failure detection for 1.572s (election complete) I0517 06:45:48.064316 16803 raft_consensus.cc:2511] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:48.064348 16486 raft_consensus.cc:2219] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0513a4bbf7d4f938b5eecba5c0967f5 in term 1. I0517 06:45:48.065212 16542 leader_election.cc:379] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [CANDIDATE]: Term 1 election: Vote granted by peer 564d4149aa01431db4135c45439f1c2a (127.4.96.3:47712) I0517 06:45:48.066231 16803 raft_consensus.cc:571] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [term 1 LEADER]: Becoming Leader. State: Replica: b0513a4bbf7d4f938b5eecba5c0967f5, State: Running, Role: LEADER I0517 06:45:48.066624 16803 consensus_queue.cc:204] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } I0517 06:45:48.071946 16113 catalog_manager.cc:3686] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5 reported cstate change: term changed from 0 to 1, leader changed from to b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4). New cstate: current_term: 1 leader_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } } I0517 06:45:48.091627 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:45:48.093802 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c6e6b14e43c24c699915460b0dcc2f61 to finish bootstrapping I0517 06:45:48.107692 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 88e96f529a8d442eacbf6b571e51c10c to finish bootstrapping W0517 06:45:47.974548 16530 tablet.cc:1678] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:48.117318 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 564d4149aa01431db4135c45439f1c2a to finish bootstrapping W0517 06:45:47.996475 16405 tablet.cc:1678] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:48.126471 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b0513a4bbf7d4f938b5eecba5c0967f5 to finish bootstrapping I0517 06:45:48.135514 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1f2902b4a6b24c379dbe0f755512f4a3 to finish bootstrapping W0517 06:45:48.144424 16654 tablet.cc:1678] T d9374f3bfb9043bdb65da277ea8671c7 P b0513a4bbf7d4f938b5eecba5c0967f5: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:48.146136 4480 test_util.cc:200] Using random seed: 796965318 I0517 06:45:48.161640 4480 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0517 06:45:48.162259 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 16532 W0517 06:45:48.180260 16094 connection.cc:512] client connection to 127.4.96.4:44014 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:44014): Connection reset by peer (error 104) W0517 06:45:48.180371 16085 connection.cc:512] client connection to 127.4.96.4:44014 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:44014): Connection reset by peer (error 104) W0517 06:45:48.181130 16095 connection.cc:512] server connection from 127.4.96.4:58638 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:58638): Connection reset by peer (error 104) W0517 06:45:48.181968 16420 connection.cc:512] server connection from 127.4.96.4:37867 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:37867): Connection reset by peer (error 104) W0517 06:45:48.182991 16291 connection.cc:512] server connection from 127.4.96.4:36451 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:36451): Connection reset by peer (error 104) W0517 06:45:48.195503 16812 meta_cache.cc:216] Tablet d9374f3bfb9043bdb65da277ea8671c7: Replica b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014) has failed: Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111) W0517 06:45:48.311609 16819 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:45:48.822304 16777 heartbeater.cc:467] Master 127.4.96.62:36173 was elected leader, sending a full tablet report... W0517 06:45:49.249650 16812 meta_cache.cc:216] Tablet d9374f3bfb9043bdb65da277ea8671c7: Replica b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014) has failed: Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111) I0517 06:45:50.133409 16872 raft_consensus.cc:435] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:50.133620 16872 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: Snoozing failure detection for 1.605s (starting election) I0517 06:45:50.133817 16872 raft_consensus.cc:457] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } I0517 06:45:50.134709 16872 leader_election.cc:243] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014) I0517 06:45:50.135119 16872 leader_election.cc:243] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 88e96f529a8d442eacbf6b571e51c10c (127.4.96.2:37572) W0517 06:45:50.137553 16416 leader_election.cc:299] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014): Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111) I0517 06:45:50.144605 16361 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d9374f3bfb9043bdb65da277ea8671c7" candidate_uuid: "564d4149aa01431db4135c45439f1c2a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88e96f529a8d442eacbf6b571e51c10c" is_pre_election: true I0517 06:45:50.144955 16361 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c: Snoozing failure detection for 1.689s (vote granted) I0517 06:45:50.145098 16361 raft_consensus.cc:2219] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 564d4149aa01431db4135c45439f1c2a in term 1. I0517 06:45:50.145764 16420 leader_election.cc:379] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [CANDIDATE]: Term 2 pre-election: Vote granted by peer 88e96f529a8d442eacbf6b571e51c10c (127.4.96.2:37572) I0517 06:45:50.145949 16420 leader_election.cc:270] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0517 06:45:50.146343 16872 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: Snoozing failure detection for 1.761s (election complete) I0517 06:45:50.146512 16872 raft_consensus.cc:2511] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 1 FOLLOWER]: Leader pre-election won for term 2 I0517 06:45:50.146668 16872 raft_consensus.cc:435] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:45:50.146806 16872 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: Snoozing failure detection for 1.731s (starting election) I0517 06:45:50.146965 16872 raft_consensus.cc:2756] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 1 FOLLOWER]: Advancing to term 2 I0517 06:45:50.149806 16872 raft_consensus.cc:457] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } I0517 06:45:50.150430 16872 leader_election.cc:243] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [CANDIDATE]: Term 2 election: Requesting vote from peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014) I0517 06:45:50.150733 16872 leader_election.cc:243] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [CANDIDATE]: Term 2 election: Requesting vote from peer 88e96f529a8d442eacbf6b571e51c10c (127.4.96.2:37572) W0517 06:45:50.151571 16416 leader_election.cc:299] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014): Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111) I0517 06:45:50.151697 16361 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d9374f3bfb9043bdb65da277ea8671c7" candidate_uuid: "564d4149aa01431db4135c45439f1c2a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88e96f529a8d442eacbf6b571e51c10c" I0517 06:45:50.152014 16361 raft_consensus.cc:2756] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c [term 1 FOLLOWER]: Advancing to term 2 I0517 06:45:50.152184 16361 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c: Snoozing failure detection for 1.627s (vote granted) I0517 06:45:50.154930 16361 raft_consensus.cc:2219] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 564d4149aa01431db4135c45439f1c2a in term 2. I0517 06:45:50.155563 16420 leader_election.cc:379] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [CANDIDATE]: Term 2 election: Vote granted by peer 88e96f529a8d442eacbf6b571e51c10c (127.4.96.2:37572) I0517 06:45:50.155716 16420 leader_election.cc:270] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0517 06:45:50.155999 16872 raft_consensus.cc:2704] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a: Snoozing failure detection for 1.821s (election complete) I0517 06:45:50.156180 16872 raft_consensus.cc:2511] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 2 FOLLOWER]: Leader election won for term 2 I0517 06:45:50.157172 16872 raft_consensus.cc:571] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [term 2 LEADER]: Becoming Leader. State: Replica: 564d4149aa01431db4135c45439f1c2a, State: Running, Role: LEADER I0517 06:45:50.157598 16872 consensus_queue.cc:204] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } I0517 06:45:50.162302 16113 catalog_manager.cc:3686] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a reported cstate change: term changed from 1 to 2, leader changed from b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4) to 564d4149aa01431db4135c45439f1c2a (127.4.96.3). New cstate: current_term: 2 leader_uuid: "564d4149aa01431db4135c45439f1c2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0513a4bbf7d4f938b5eecba5c0967f5" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 44014 } } peers { permanent_uuid: "564d4149aa01431db4135c45439f1c2a" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 47712 } } peers { permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 } } } I0517 06:45:50.245905 16361 raft_consensus.cc:1057] T d9374f3bfb9043bdb65da277ea8671c7 P 88e96f529a8d442eacbf6b571e51c10c [term 2 FOLLOWER]: Refusing update from remote peer 564d4149aa01431db4135c45439f1c2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch) W0517 06:45:50.246160 16416 consensus_peers.cc:434] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a -> Peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014): Couldn't send request to peer b0513a4bbf7d4f938b5eecba5c0967f5 for tablet d9374f3bfb9043bdb65da277ea8671c7. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0517 06:45:50.247222 16875 consensus_queue.cc:963] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "88e96f529a8d442eacbf6b571e51c10c" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 37572 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0517 06:45:50.641340 16416 consensus_peers.cc:434] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a -> Peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014): Couldn't send request to peer b0513a4bbf7d4f938b5eecba5c0967f5 for tablet d9374f3bfb9043bdb65da277ea8671c7. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0517 06:45:51.207486 16416 consensus_peers.cc:434] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a -> Peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014): Couldn't send request to peer b0513a4bbf7d4f938b5eecba5c0967f5 for tablet d9374f3bfb9043bdb65da277ea8671c7. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0517 06:45:51.718945 16416 consensus_peers.cc:434] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a -> Peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014): Couldn't send request to peer b0513a4bbf7d4f938b5eecba5c0967f5 for tablet d9374f3bfb9043bdb65da277ea8671c7. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0517 06:45:52.312839 16416 consensus_peers.cc:434] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a -> Peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014): Couldn't send request to peer b0513a4bbf7d4f938b5eecba5c0967f5 for tablet d9374f3bfb9043bdb65da277ea8671c7. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0517 06:45:52.902752 16416 consensus_peers.cc:434] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a -> Peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014): Couldn't send request to peer b0513a4bbf7d4f938b5eecba5c0967f5 for tablet d9374f3bfb9043bdb65da277ea8671c7. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0517 06:45:53.465656 16416 consensus_peers.cc:434] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a -> Peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014): Couldn't send request to peer b0513a4bbf7d4f938b5eecba5c0967f5 for tablet d9374f3bfb9043bdb65da277ea8671c7. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W0517 06:45:53.881027 16416 consensus_peers.cc:434] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a -> Peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014): Couldn't send request to peer b0513a4bbf7d4f938b5eecba5c0967f5 for tablet d9374f3bfb9043bdb65da277ea8671c7. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. Corruption: ksck pre-move health check failed: timed out with ksck errors remaining: last error: 1 out of 1 table(s) are not healthy W0517 06:45:54.502342 16416 consensus_peers.cc:434] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a -> Peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014): Couldn't send request to peer b0513a4bbf7d4f938b5eecba5c0967f5 for tablet d9374f3bfb9043bdb65da277ea8671c7. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. W0517 06:45:55.130625 16416 consensus_peers.cc:434] T d9374f3bfb9043bdb65da277ea8671c7 P 564d4149aa01431db4135c45439f1c2a -> Peer b0513a4bbf7d4f938b5eecba5c0967f5 (127.4.96.4:44014): Couldn't send request to peer b0513a4bbf7d4f938b5eecba5c0967f5 for tablet d9374f3bfb9043bdb65da277ea8671c7. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.4:44014: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. I0517 06:45:55.132635 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 16148 I0517 06:45:55.152756 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 16283 W0517 06:45:55.152741 16085 connection.cc:512] client connection to 127.4.96.1:38880 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:38880): Connection reset by peer (error 104) W0517 06:45:55.154114 16095 connection.cc:512] server connection from 127.4.96.1:42398 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:42398): Connection reset by peer (error 104) W0517 06:45:55.189456 16085 connection.cc:512] client connection to 127.4.96.2:37572 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:37572): Connection reset by peer (error 104) W0517 06:45:55.189466 16098 connection.cc:512] client connection to 127.4.96.2:37572 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:37572): Connection reset by peer (error 104) W0517 06:45:55.190115 16420 connection.cc:512] client connection to 127.4.96.2:37572 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:37572): Connection reset by peer (error 104) W0517 06:45:55.191577 16095 connection.cc:512] server connection from 127.4.96.2:33855 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:33855): Connection reset by peer (error 104) I0517 06:45:55.198153 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 16408 W0517 06:45:55.229672 16085 connection.cc:512] client connection to 127.4.96.3:47712 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:47712): Connection reset by peer (error 104) W0517 06:45:55.230207 16096 connection.cc:512] client connection to 127.4.96.3:47712 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:47712): Connection reset by peer (error 104) I0517 06:45:55.230798 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 16656 W0517 06:45:55.231696 16094 connection.cc:512] server connection from 127.4.96.3:36935 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:36935): Connection reset by peer (error 104) W0517 06:45:55.252579 16085 connection.cc:512] client connection to 127.4.96.5:41271 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:41271): Connection reset by peer (error 104) W0517 06:45:55.253751 16096 connection.cc:512] server connection from 127.4.96.5:48217 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:48217): Connection reset by peer (error 104) I0517 06:45:55.253831 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 16086 W0517 06:45:55.278874 16085 connection.cc:512] client connection to 127.4.96.62:36173 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:36173): Connection reset by peer (error 104) [ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/1 (10400 ms) [ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/2 I0517 06:45:55.297442 4480 test_util.cc:200] Using random seed: 804116620 I0517 06:45:55.300515 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:45:55.300611 4480 ts_itest-base.cc:114] -------------- I0517 06:45:55.300683 4480 ts_itest-base.cc:115] 5 tablet servers I0517 06:45:55.300755 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:45:55.300825 4480 ts_itest-base.cc:117] -------------- I0517 06:45:55.302819 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:36482 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:55.418825 16985 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:55.419147 16985 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:55.428058 16985 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:55.428210 16985 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:45:55.428308 16985 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:55.428429 16985 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:45:55.428584 16985 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:45:55.442831 16985 master_main.cc:73] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:36482 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:55.450515 16985 master_main.cc:80] Initializing master server... I0517 06:45:55.452021 16985 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 176819us I0517 06:45:55.452759 16985 fs_manager.cc:260] Metadata directory not provided I0517 06:45:55.452875 16985 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:55.453068 16985 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:45:55.453177 16985 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:55.460448 16985 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "470e619f81f44ddf8b97627bce30362f" format_stamp: "Formatted at 2018-05-17 06:45:55 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:55.461109 16985 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "470e619f81f44ddf8b97627bce30362f" format_stamp: "Formatted at 2018-05-17 06:45:55 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:55.470360 16985 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:45:55.471449 16985 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:55.471846 16985 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:55.478170 16985 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:45:55.478276 16985 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "470e619f81f44ddf8b97627bce30362f" format_stamp: "Formatted at 2018-05-17 06:45:55 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:55.478500 16985 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:55.643831 16985 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:55.645509 16985 master_main.cc:83] Starting Master server... I0517 06:45:55.685284 16985 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:36482 I0517 06:45:55.686118 16985 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:45:55.686215 16985 webserver.cc:184] Document root disabled I0517 06:45:55.687417 16985 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:34332/ I0517 06:45:55.688099 16985 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:45:55.690631 17046 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:55.697815 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 16985 I0517 06:45:55.699151 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36482 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:55.707279 17046 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f: Bootstrap starting. I0517 06:45:55.709486 17046 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f: No blocks or log segments found. Creating new log. I0517 06:45:55.710654 17046 log.cc:520] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f: Log is configured to *not* fsync() on all Append() calls I0517 06:45:55.713007 17046 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f: No bootstrap required, opened a new log I0517 06:45:55.724797 17046 raft_consensus.cc:303] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470e619f81f44ddf8b97627bce30362f" member_type: VOTER } I0517 06:45:55.725100 17046 raft_consensus.cc:329] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:55.725225 17046 raft_consensus.cc:604] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 470e619f81f44ddf8b97627bce30362f, State: Initialized, Role: FOLLOWER I0517 06:45:55.725625 17046 consensus_queue.cc:227] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470e619f81f44ddf8b97627bce30362f" member_type: VOTER } I0517 06:45:55.725944 17046 raft_consensus.cc:343] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:45:55.726080 17046 raft_consensus.cc:435] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:45:55.726222 17046 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f: Snoozing failure detection for 1.660s (starting election) I0517 06:45:55.726356 17046 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:55.728706 17046 raft_consensus.cc:457] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470e619f81f44ddf8b97627bce30362f" member_type: VOTER } I0517 06:45:55.729109 17046 leader_election.cc:270] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:55.730657 17049 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f: Snoozing failure detection for 1.881s (election complete) I0517 06:45:55.730934 17049 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:55.737586 17049 raft_consensus.cc:571] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [term 1 LEADER]: Becoming Leader. State: Replica: 470e619f81f44ddf8b97627bce30362f, State: Running, Role: LEADER I0517 06:45:55.738001 17049 consensus_queue.cc:204] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470e619f81f44ddf8b97627bce30362f" member_type: VOTER } I0517 06:45:55.741489 17046 sys_catalog.cc:335] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "470e619f81f44ddf8b97627bce30362f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470e619f81f44ddf8b97627bce30362f" member_type: VOTER } } I0517 06:45:55.741904 17046 sys_catalog.cc:338] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [sys.catalog]: This master's current role is: LEADER I0517 06:45:55.741722 17051 sys_catalog.cc:335] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 470e619f81f44ddf8b97627bce30362f. Latest consensus state: current_term: 1 leader_uuid: "470e619f81f44ddf8b97627bce30362f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470e619f81f44ddf8b97627bce30362f" member_type: VOTER } } I0517 06:45:55.801760 17051 sys_catalog.cc:338] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [sys.catalog]: This master's current role is: LEADER I0517 06:45:55.743005 17050 sys_catalog.cc:335] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "470e619f81f44ddf8b97627bce30362f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "470e619f81f44ddf8b97627bce30362f" member_type: VOTER } } I0517 06:45:55.808640 17050 sys_catalog.cc:338] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [sys.catalog]: This master's current role is: LEADER I0517 06:45:55.809329 17046 sys_catalog.cc:422] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f [sys.catalog]: configured and running, proceeding with master startup. I0517 06:45:55.809561 17053 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:45:55.811022 16985 master_main.cc:86] Master server successfully started. I0517 06:45:55.814453 17053 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:55.846945 17047 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:55.847443 17047 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:45:55.857488 17053 catalog_manager.cc:899] Generated new certificate authority record I0517 06:45:55.858927 17053 catalog_manager.cc:1021] Loading token signing keys... W0517 06:45:55.861620 17047 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:55.861858 17047 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:55.862362 17047 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:45:55.867867 17053 catalog_manager.cc:3962] T 00000000000000000000000000000000 P 470e619f81f44ddf8b97627bce30362f: Generated new TSK 0 I0517 06:45:55.891309 17047 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36482 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:55.904777 17047 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:55.907248 17047 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 176819us I0517 06:45:55.907856 17047 fs_manager.cc:260] Metadata directory not provided I0517 06:45:55.908025 17047 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:55.908309 17047 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:45:55.908475 17047 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:55.921305 17047 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "cfd3f59245e74fbe9e3edad3e130e9f2" format_stamp: "Formatted at 2018-05-17 06:45:55 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:55.922294 17047 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "cfd3f59245e74fbe9e3edad3e130e9f2" format_stamp: "Formatted at 2018-05-17 06:45:55 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:55.947063 17047 fs_manager.cc:495] Time spent creating directory manager: real 0.024s user 0.008s sys 0.016s I0517 06:45:55.948210 17047 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:55.948606 17047 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:55.959085 17047 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.008s sys 0.004s I0517 06:45:55.959211 17047 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "cfd3f59245e74fbe9e3edad3e130e9f2" format_stamp: "Formatted at 2018-05-17 06:45:55 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:55.959412 17047 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:56.217033 17047 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:56.230974 17047 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:56.242254 17047 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:56.680196 17047 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:55588 I0517 06:45:56.681071 17047 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:45:56.681195 17047 webserver.cc:184] Document root disabled I0517 06:45:56.682561 17047 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:44891/ I0517 06:45:56.683220 17047 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:45:56.688329 17047 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:56.692778 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 17047 I0517 06:45:56.693696 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36482 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:56.710414 17178 heartbeater.cc:346] Connected to a master server at 127.4.96.62:36482 I0517 06:45:56.710669 17178 heartbeater.cc:426] Registering TS with master... I0517 06:45:56.711115 17178 heartbeater.cc:475] Master 127.4.96.62:36482 requested a full tablet report, sending... I0517 06:45:56.712620 17013 ts_manager.cc:81] Registered new tserver with Master: cfd3f59245e74fbe9e3edad3e130e9f2 (127.4.96.1:55588) I0517 06:45:56.714354 17013 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:50433 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:56.851296 17182 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:56.851768 17182 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:56.865911 17182 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:56.866158 17182 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:56.866641 17182 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:45:56.887046 17182 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36482 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:56.900321 17182 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:56.902789 17182 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 177319us I0517 06:45:56.903410 17182 fs_manager.cc:260] Metadata directory not provided I0517 06:45:56.903580 17182 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:56.903865 17182 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:45:56.904036 17182 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:56.916832 17182 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" format_stamp: "Formatted at 2018-05-17 06:45:56 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:56.917779 17182 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" format_stamp: "Formatted at 2018-05-17 06:45:56 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:56.931746 17182 fs_manager.cc:495] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s I0517 06:45:56.933378 17182 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:56.933914 17182 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:56.944582 17182 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:45:56.944738 17182 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" format_stamp: "Formatted at 2018-05-17 06:45:56 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:56.945000 17182 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:57.106526 17182 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:57.109625 17182 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:57.110703 17182 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:57.233289 17182 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:40420 I0517 06:45:57.234176 17182 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:45:57.234292 17182 webserver.cc:184] Document root disabled I0517 06:45:57.235507 17182 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:51987/ I0517 06:45:57.236199 17182 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:45:57.242863 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 17182 I0517 06:45:57.243791 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36482 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:57.254878 17182 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:57.270696 17303 heartbeater.cc:346] Connected to a master server at 127.4.96.62:36482 I0517 06:45:57.270933 17303 heartbeater.cc:426] Registering TS with master... I0517 06:45:57.271365 17303 heartbeater.cc:475] Master 127.4.96.62:36482 requested a full tablet report, sending... I0517 06:45:57.272673 17013 ts_manager.cc:81] Registered new tserver with Master: 7cd36755c08c4fa7bc42cb8b4d9be9cf (127.4.96.2:40420) I0517 06:45:57.273757 17013 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:53271 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:57.355515 17307 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:57.355846 17307 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:57.363680 17307 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:57.363837 17307 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:57.364126 17307 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:45:57.376677 17307 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36482 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:57.384635 17307 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:57.386378 17307 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 177819us I0517 06:45:57.386792 17307 fs_manager.cc:260] Metadata directory not provided I0517 06:45:57.386898 17307 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:57.387076 17307 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:45:57.387177 17307 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:57.394824 17307 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "5da82191918541a0ba8e299970b150a0" format_stamp: "Formatted at 2018-05-17 06:45:57 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:57.395450 17307 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "5da82191918541a0ba8e299970b150a0" format_stamp: "Formatted at 2018-05-17 06:45:57 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:57.404747 17307 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:45:57.405850 17307 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:57.406239 17307 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:57.412468 17307 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:45:57.412580 17307 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "5da82191918541a0ba8e299970b150a0" format_stamp: "Formatted at 2018-05-17 06:45:57 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:57.412761 17307 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:57.566247 17307 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:57.568536 17307 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:57.569453 17307 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:57.656625 17307 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:54414 I0517 06:45:57.657867 17307 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:45:57.658023 17307 webserver.cc:184] Document root disabled I0517 06:45:57.659232 17307 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:56103/ I0517 06:45:57.659884 17307 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:45:57.662348 17307 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:57.663746 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 17307 I0517 06:45:57.665021 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36482 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:57.683372 17428 heartbeater.cc:346] Connected to a master server at 127.4.96.62:36482 I0517 06:45:57.683610 17428 heartbeater.cc:426] Registering TS with master... I0517 06:45:57.684025 17428 heartbeater.cc:475] Master 127.4.96.62:36482 requested a full tablet report, sending... I0517 06:45:57.685370 17013 ts_manager.cc:81] Registered new tserver with Master: 5da82191918541a0ba8e299970b150a0 (127.4.96.3:54414) I0517 06:45:57.686489 17013 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:35893 I0517 06:45:57.716652 17178 heartbeater.cc:467] Master 127.4.96.62:36482 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:57.773540 17431 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:57.773849 17431 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:57.781599 17431 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:57.781754 17431 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:57.782052 17431 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:45:57.794584 17431 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36482 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:57.802588 17431 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:57.804353 17431 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 177819us I0517 06:45:57.804764 17431 fs_manager.cc:260] Metadata directory not provided I0517 06:45:57.804883 17431 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:57.805069 17431 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:45:57.805173 17431 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:57.812711 17431 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "68873422d063453797c080d8192b33be" format_stamp: "Formatted at 2018-05-17 06:45:57 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:57.813354 17431 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "68873422d063453797c080d8192b33be" format_stamp: "Formatted at 2018-05-17 06:45:57 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:57.822680 17431 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:45:57.823810 17431 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:57.824214 17431 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:57.830482 17431 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:45:57.830588 17431 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "68873422d063453797c080d8192b33be" format_stamp: "Formatted at 2018-05-17 06:45:57 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:57.830798 17431 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:57.981112 17431 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:57.983455 17431 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:57.984351 17431 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:58.069571 17431 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:47104 I0517 06:45:58.070464 17431 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:45:58.070574 17431 webserver.cc:184] Document root disabled I0517 06:45:58.071821 17431 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:32783/ I0517 06:45:58.072502 17431 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:45:58.074997 17431 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:58.080545 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 17431 I0517 06:45:58.081807 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.4.96.5:0 --local_ip_for_outbound_sockets=127.4.96.5 --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36482 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0517 06:45:58.094869 17552 heartbeater.cc:346] Connected to a master server at 127.4.96.62:36482 I0517 06:45:58.095095 17552 heartbeater.cc:426] Registering TS with master... I0517 06:45:58.095520 17552 heartbeater.cc:475] Master 127.4.96.62:36482 requested a full tablet report, sending... I0517 06:45:58.096885 17013 ts_manager.cc:81] Registered new tserver with Master: 68873422d063453797c080d8192b33be (127.4.96.4:47104) I0517 06:45:58.098048 17013 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:44712 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:45:58.191071 17555 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:45:58.191401 17555 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:45:58.199273 17555 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0517 06:45:58.199446 17555 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:45:58.199753 17555 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.5 I0517 06:45:58.212304 17555 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:36482 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.5 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:45:58.220366 17555 tablet_server_main.cc:85] Initializing tablet server... I0517 06:45:58.222162 17555 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 178319us I0517 06:45:58.222590 17555 fs_manager.cc:260] Metadata directory not provided I0517 06:45:58.223366 17555 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:45:58.223587 17555 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0517 06:45:58.223704 17555 server_base.cc:433] Attempting to create new FS layout instead I0517 06:45:58.231241 17555 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "4d7de5247d5c4bebb8312c873b9f10b7" format_stamp: "Formatted at 2018-05-17 06:45:58 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:58.231885 17555 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "4d7de5247d5c4bebb8312c873b9f10b7" format_stamp: "Formatted at 2018-05-17 06:45:58 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:58.241294 17555 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:45:58.242460 17555 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:45:58.242864 17555 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:45:58.249199 17555 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:45:58.249305 17555 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal uuid: "4d7de5247d5c4bebb8312c873b9f10b7" format_stamp: "Formatted at 2018-05-17 06:45:58 on dist-test-slave-dist-test-slave-26mv" I0517 06:45:58.249516 17555 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:45:58.276043 17303 heartbeater.cc:467] Master 127.4.96.62:36482 was elected leader, sending a full tablet report... I0517 06:45:58.396112 17555 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:45:58.398434 17555 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:45:58.399371 17555 tablet_server_main.cc:90] Starting tablet server... I0517 06:45:58.483988 17555 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.5:35896 I0517 06:45:58.484875 17555 webserver.cc:173] Starting webserver on 127.4.96.5:0 I0517 06:45:58.484982 17555 webserver.cc:184] Document root disabled I0517 06:45:58.486217 17555 webserver.cc:311] Webserver started. Bound to: http://127.4.96.5:59931/ I0517 06:45:58.486905 17555 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb I0517 06:45:58.489630 17555 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:45:58.490161 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 17555 I0517 06:45:58.511839 17676 heartbeater.cc:346] Connected to a master server at 127.4.96.62:36482 I0517 06:45:58.512086 17676 heartbeater.cc:426] Registering TS with master... I0517 06:45:58.512493 17676 heartbeater.cc:475] Master 127.4.96.62:36482 requested a full tablet report, sending... I0517 06:45:58.513739 17013 ts_manager.cc:81] Registered new tserver with Master: 4d7de5247d5c4bebb8312c873b9f10b7 (127.4.96.5:35896) I0517 06:45:58.515167 17013 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.5:56815 I0517 06:45:58.515967 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:45:58.532629 17013 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42292: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:45:58.572198 17489 tablet_service.cc:759] Processing CreateTablet for tablet 0b8b163cec1348908d114141f19af288 (table=TestTable [id=71121150ba2b48b88f708e1ae44990cc]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:58.573093 17489 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0b8b163cec1348908d114141f19af288. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:58.577419 17240 tablet_service.cc:759] Processing CreateTablet for tablet 0b8b163cec1348908d114141f19af288 (table=TestTable [id=71121150ba2b48b88f708e1ae44990cc]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:58.578327 17240 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0b8b163cec1348908d114141f19af288. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:58.578671 17365 tablet_service.cc:759] Processing CreateTablet for tablet 0b8b163cec1348908d114141f19af288 (table=TestTable [id=71121150ba2b48b88f708e1ae44990cc]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:45:58.579493 17365 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0b8b163cec1348908d114141f19af288. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:45:58.589181 17489 ts_tablet_manager.cc:1173] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:58.589648 17240 ts_tablet_manager.cc:1173] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:58.595984 17365 ts_tablet_manager.cc:1173] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: Registered tablet (data state: TABLET_DATA_READY) I0517 06:45:58.597674 17696 ts_tablet_manager.cc:1028] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: Bootstrapping tablet I0517 06:45:58.598381 17698 ts_tablet_manager.cc:1028] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: Bootstrapping tablet I0517 06:45:58.598632 17697 ts_tablet_manager.cc:1028] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf: Bootstrapping tablet I0517 06:45:58.599195 17697 tablet_bootstrap.cc:436] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf: Bootstrap starting. I0517 06:45:58.599459 17696 tablet_bootstrap.cc:436] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: Bootstrap starting. I0517 06:45:58.600262 17698 tablet_bootstrap.cc:436] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: Bootstrap starting. I0517 06:45:58.601843 17697 tablet_bootstrap.cc:587] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf: No blocks or log segments found. Creating new log. I0517 06:45:58.602430 17696 tablet_bootstrap.cc:587] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: No blocks or log segments found. Creating new log. I0517 06:45:58.603152 17698 tablet_bootstrap.cc:587] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: No blocks or log segments found. Creating new log. I0517 06:45:58.603268 17697 log.cc:520] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf: Log is configured to *not* fsync() on all Append() calls I0517 06:45:58.603969 17696 log.cc:520] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: Log is configured to *not* fsync() on all Append() calls I0517 06:45:58.604770 17698 log.cc:520] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: Log is configured to *not* fsync() on all Append() calls I0517 06:45:58.606981 17696 tablet_bootstrap.cc:436] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: No bootstrap required, opened a new log I0517 06:45:58.607368 17696 ts_tablet_manager.cc:1045] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s I0517 06:45:58.607494 17698 tablet_bootstrap.cc:436] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: No bootstrap required, opened a new log I0517 06:45:58.607760 17698 ts_tablet_manager.cc:1045] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s I0517 06:45:58.607971 17697 tablet_bootstrap.cc:436] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf: No bootstrap required, opened a new log I0517 06:45:58.608306 17697 ts_tablet_manager.cc:1045] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s I0517 06:45:58.618225 17698 raft_consensus.cc:303] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } I0517 06:45:58.618795 17698 raft_consensus.cc:329] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:58.618971 17698 raft_consensus.cc:604] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da82191918541a0ba8e299970b150a0, State: Initialized, Role: FOLLOWER I0517 06:45:58.619488 17698 consensus_queue.cc:227] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } I0517 06:45:58.622838 17696 raft_consensus.cc:303] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } I0517 06:45:58.623582 17696 raft_consensus.cc:329] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:58.623761 17696 raft_consensus.cc:604] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68873422d063453797c080d8192b33be, State: Initialized, Role: FOLLOWER I0517 06:45:58.624389 17696 consensus_queue.cc:227] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } I0517 06:45:58.624857 17428 heartbeater.cc:467] Master 127.4.96.62:36482 was elected leader, sending a full tablet report... I0517 06:45:58.625118 17697 raft_consensus.cc:303] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } I0517 06:45:58.625859 17697 raft_consensus.cc:329] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:45:58.626052 17697 raft_consensus.cc:604] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cd36755c08c4fa7bc42cb8b4d9be9cf, State: Initialized, Role: FOLLOWER I0517 06:45:58.627738 17697 consensus_queue.cc:227] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } I0517 06:45:58.633730 17552 heartbeater.cc:467] Master 127.4.96.62:36482 was elected leader, sending a full tablet report... I0517 06:45:58.635301 17696 ts_tablet_manager.cc:1073] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: Time spent starting tablet: real 0.028s user 0.020s sys 0.004s I0517 06:45:58.636349 17697 ts_tablet_manager.cc:1073] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf: Time spent starting tablet: real 0.028s user 0.020s sys 0.004s I0517 06:45:58.625319 17698 ts_tablet_manager.cc:1073] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: Time spent starting tablet: real 0.017s user 0.016s sys 0.004s W0517 06:45:58.664412 17429 tablet.cc:1678] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:58.731948 17703 raft_consensus.cc:435] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:58.793177 17703 raft_consensus.cc:2704] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: Snoozing failure detection for 1.709s (starting election) I0517 06:45:58.793500 17703 raft_consensus.cc:457] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } I0517 06:45:58.794844 17703 leader_election.cc:243] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 5da82191918541a0ba8e299970b150a0 (127.4.96.3:54414) I0517 06:45:58.795419 17703 leader_election.cc:243] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 7cd36755c08c4fa7bc42cb8b4d9be9cf (127.4.96.2:40420) I0517 06:45:58.810071 17385 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0b8b163cec1348908d114141f19af288" candidate_uuid: "68873422d063453797c080d8192b33be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da82191918541a0ba8e299970b150a0" is_pre_election: true I0517 06:45:58.810667 17385 raft_consensus.cc:2704] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: Snoozing failure detection for 1.977s (vote granted) I0517 06:45:58.810871 17385 raft_consensus.cc:2219] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68873422d063453797c080d8192b33be in term 0. I0517 06:45:58.811084 17260 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0b8b163cec1348908d114141f19af288" candidate_uuid: "68873422d063453797c080d8192b33be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" is_pre_election: true I0517 06:45:58.811759 17441 leader_election.cc:379] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [CANDIDATE]: Term 1 pre-election: Vote granted by peer 5da82191918541a0ba8e299970b150a0 (127.4.96.3:54414) I0517 06:45:58.811962 17441 leader_election.cc:270] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:45:58.812430 17703 raft_consensus.cc:2704] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: Snoozing failure detection for 2.006s (election complete) I0517 06:45:58.812670 17703 raft_consensus.cc:2511] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:45:58.812866 17703 raft_consensus.cc:435] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:45:58.813043 17703 raft_consensus.cc:2704] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: Snoozing failure detection for 1.669s (starting election) I0517 06:45:58.813220 17703 raft_consensus.cc:2756] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:58.817425 17703 raft_consensus.cc:457] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } I0517 06:45:58.818337 17703 leader_election.cc:243] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [CANDIDATE]: Term 1 election: Requesting vote from peer 5da82191918541a0ba8e299970b150a0 (127.4.96.3:54414) I0517 06:45:58.818727 17703 leader_election.cc:243] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [CANDIDATE]: Term 1 election: Requesting vote from peer 7cd36755c08c4fa7bc42cb8b4d9be9cf (127.4.96.2:40420) I0517 06:45:58.819453 17385 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0b8b163cec1348908d114141f19af288" candidate_uuid: "68873422d063453797c080d8192b33be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da82191918541a0ba8e299970b150a0" I0517 06:45:58.819793 17385 raft_consensus.cc:2756] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:58.820009 17385 raft_consensus.cc:2704] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: Snoozing failure detection for 1.765s (vote granted) I0517 06:45:58.822824 17385 raft_consensus.cc:2219] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68873422d063453797c080d8192b33be in term 1. I0517 06:45:58.823485 17441 leader_election.cc:379] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [CANDIDATE]: Term 1 election: Vote granted by peer 5da82191918541a0ba8e299970b150a0 (127.4.96.3:54414) I0517 06:45:58.823654 17441 leader_election.cc:270] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:45:58.824007 17703 raft_consensus.cc:2704] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: Snoozing failure detection for 1.548s (election complete) I0517 06:45:58.824288 17703 raft_consensus.cc:2511] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:45:58.825687 17703 raft_consensus.cc:571] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 1 LEADER]: Becoming Leader. State: Replica: 68873422d063453797c080d8192b33be, State: Running, Role: LEADER I0517 06:45:58.826212 17703 consensus_queue.cc:204] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } W0517 06:45:58.827054 17553 tablet.cc:1678] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:45:58.762889 17304 tablet.cc:1678] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:45:58.831995 17013 catalog_manager.cc:3686] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be reported cstate change: term changed from 0 to 1, leader changed from to 68873422d063453797c080d8192b33be (127.4.96.4). New cstate: current_term: 1 leader_uuid: "68873422d063453797c080d8192b33be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } } I0517 06:45:58.768712 17704 raft_consensus.cc:435] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:45:58.851873 17704 raft_consensus.cc:2704] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf: Snoozing failure detection for 1.752s (starting election) I0517 06:45:58.852169 17704 raft_consensus.cc:457] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } I0517 06:45:58.853163 17704 leader_election.cc:243] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 5da82191918541a0ba8e299970b150a0 (127.4.96.3:54414) I0517 06:45:58.853260 17260 raft_consensus.cc:2704] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf: Snoozing failure detection for 1.993s (vote granted) I0517 06:45:58.853540 17260 raft_consensus.cc:2219] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68873422d063453797c080d8192b33be in term 0. I0517 06:45:58.853595 17704 leader_election.cc:243] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 68873422d063453797c080d8192b33be (127.4.96.4:47104) I0517 06:45:58.856493 17439 leader_election.cc:379] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [CANDIDATE]: Term 1 pre-election: Vote granted by peer 7cd36755c08c4fa7bc42cb8b4d9be9cf (127.4.96.2:40420) I0517 06:45:58.820250 17259 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0b8b163cec1348908d114141f19af288" candidate_uuid: "68873422d063453797c080d8192b33be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" I0517 06:45:58.862251 17259 raft_consensus.cc:2756] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 0 FOLLOWER]: Advancing to term 1 I0517 06:45:58.862486 17259 raft_consensus.cc:2704] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf: Snoozing failure detection for 2.119s (vote granted) I0517 06:45:58.867626 17259 raft_consensus.cc:2219] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68873422d063453797c080d8192b33be in term 1. I0517 06:45:58.869168 17439 leader_election.cc:379] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [CANDIDATE]: Term 1 election: Vote granted by peer 7cd36755c08c4fa7bc42cb8b4d9be9cf (127.4.96.2:40420) I0517 06:45:58.871577 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:45:58.874339 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cfd3f59245e74fbe9e3edad3e130e9f2 to finish bootstrapping I0517 06:45:58.876492 17509 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0b8b163cec1348908d114141f19af288" candidate_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68873422d063453797c080d8192b33be" is_pre_election: true I0517 06:45:58.877012 17509 raft_consensus.cc:2180] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 7cd36755c08c4fa7bc42cb8b4d9be9cf for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:45:58.878280 17194 leader_election.cc:393] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [CANDIDATE]: Term 1 pre-election: Vote denied by peer 68873422d063453797c080d8192b33be (127.4.96.4:47104). Message: Invalid argument: T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 7cd36755c08c4fa7bc42cb8b4d9be9cf for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:45:58.879735 17385 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0b8b163cec1348908d114141f19af288" candidate_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da82191918541a0ba8e299970b150a0" is_pre_election: true I0517 06:45:58.880146 17385 raft_consensus.cc:2149] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7cd36755c08c4fa7bc42cb8b4d9be9cf in current term 1: Already voted for candidate 68873422d063453797c080d8192b33be in this term. I0517 06:45:58.880939 17192 leader_election.cc:393] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [CANDIDATE]: Term 1 pre-election: Vote denied by peer 5da82191918541a0ba8e299970b150a0 (127.4.96.3:54414). Message: Invalid argument: T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7cd36755c08c4fa7bc42cb8b4d9be9cf in current term 1: Already voted for candidate 68873422d063453797c080d8192b33be in this term. I0517 06:45:58.881135 17192 leader_election.cc:270] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0517 06:45:58.881541 17704 raft_consensus.cc:2704] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf: Snoozing failure detection for 1.919s (election complete) I0517 06:45:58.881772 17704 raft_consensus.cc:2459] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0517 06:45:58.887325 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7cd36755c08c4fa7bc42cb8b4d9be9cf to finish bootstrapping I0517 06:45:58.895957 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5da82191918541a0ba8e299970b150a0 to finish bootstrapping I0517 06:45:58.904322 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 68873422d063453797c080d8192b33be to finish bootstrapping I0517 06:45:58.912770 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4d7de5247d5c4bebb8312c873b9f10b7 to finish bootstrapping I0517 06:45:58.923153 4480 test_util.cc:200] Using random seed: 807742334 I0517 06:45:58.938921 4480 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0517 06:45:58.939734 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 17047 W0517 06:45:58.961562 16984 connection.cc:512] client connection to 127.4.96.1:55588 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:55588): Connection reset by peer (error 104) I0517 06:45:58.961942 17259 raft_consensus.cc:1057] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 1 FOLLOWER]: Refusing update from remote peer 68873422d063453797c080d8192b33be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0517 06:45:58.962496 17385 raft_consensus.cc:1057] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [term 1 FOLLOWER]: Refusing update from remote peer 68873422d063453797c080d8192b33be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) W0517 06:45:58.962877 16994 connection.cc:512] server connection from 127.4.96.1:50433 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:50433): Connection reset by peer (error 104) I0517 06:45:58.963646 17703 consensus_queue.cc:963] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:45:58.964222 17707 consensus_queue.cc:963] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0517 06:45:59.154165 17723 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:45:59.517823 17676 heartbeater.cc:467] Master 127.4.96.62:36482 was elected leader, sending a full tablet report... I0517 06:45:59.709589 17509 consensus_peers.cc:456] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be -> Peer 7cd36755c08c4fa7bc42cb8b4d9be9cf (127.4.96.2:40420): Closing peer: 7cd36755c08c4fa7bc42cb8b4d9be9cf I0517 06:45:59.709825 17509 consensus_peers.cc:456] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be -> Peer 5da82191918541a0ba8e299970b150a0 (127.4.96.3:54414): Closing peer: 5da82191918541a0ba8e299970b150a0 I0517 06:45:59.710234 17509 consensus_queue.cc:204] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } peers { permanent_uuid: "4d7de5247d5c4bebb8312c873b9f10b7" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 35896 } } I0517 06:45:59.714601 17259 raft_consensus.cc:1057] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 1 FOLLOWER]: Refusing update from remote peer 68873422d063453797c080d8192b33be: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch) I0517 06:45:59.715423 17384 raft_consensus.cc:1057] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [term 1 FOLLOWER]: Refusing update from remote peer 68873422d063453797c080d8192b33be: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch) I0517 06:45:59.715670 17774 consensus_queue.cc:963] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 66, Time since last communication: 0.000s I0517 06:45:59.716418 17703 consensus_queue.cc:963] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 66, Time since last communication: 0.000s I0517 06:45:59.720640 17774 raft_consensus.cc:2658] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index -1 to 68, VOTER 4d7de5247d5c4bebb8312c873b9f10b7 (127.4.96.5) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } peers { permanent_uuid: "4d7de5247d5c4bebb8312c873b9f10b7" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 35896 } } } I0517 06:45:59.721741 17384 raft_consensus.cc:2658] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index -1 to 68, VOTER 4d7de5247d5c4bebb8312c873b9f10b7 (127.4.96.5) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } peers { permanent_uuid: "4d7de5247d5c4bebb8312c873b9f10b7" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 35896 } } } W0517 06:45:59.729857 17440 consensus_peers.cc:434] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be -> Peer 4d7de5247d5c4bebb8312c873b9f10b7 (127.4.96.5:35896): Couldn't send request to peer 4d7de5247d5c4bebb8312c873b9f10b7 for tablet 0b8b163cec1348908d114141f19af288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b8b163cec1348908d114141f19af288. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:45:59.729387 17013 catalog_manager.cc:3686] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be reported cstate change: config changed from index -1 to 68, VOTER 4d7de5247d5c4bebb8312c873b9f10b7 (127.4.96.5) added. New cstate: current_term: 1 leader_uuid: "68873422d063453797c080d8192b33be" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } peers { permanent_uuid: "4d7de5247d5c4bebb8312c873b9f10b7" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 35896 } } } I0517 06:45:59.734203 17259 raft_consensus.cc:2658] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index -1 to 68, VOTER 4d7de5247d5c4bebb8312c873b9f10b7 (127.4.96.5) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } peers { permanent_uuid: "4d7de5247d5c4bebb8312c873b9f10b7" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 35896 } } } I0517 06:46:00.252149 17811 ts_tablet_manager.cc:669] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7: Initiating tablet copy from peer 68873422d063453797c080d8192b33be (127.4.96.4:47104) I0517 06:46:00.252544 17811 tablet_copy_client.cc:239] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.4:47104 I0517 06:46:00.263926 17529 tablet_copy_service.cc:133] P 68873422d063453797c080d8192b33be: Received BeginTabletCopySession request for tablet 0b8b163cec1348908d114141f19af288 from peer 4d7de5247d5c4bebb8312c873b9f10b7 ({username='slave'} at 127.4.96.5:39958) I0517 06:46:00.264245 17529 tablet_copy_service.cc:154] P 68873422d063453797c080d8192b33be: Beginning new tablet copy session on tablet 0b8b163cec1348908d114141f19af288 from peer 4d7de5247d5c4bebb8312c873b9f10b7 at {username='slave'} at 127.4.96.5:39958: session id = 4d7de5247d5c4bebb8312c873b9f10b7-0b8b163cec1348908d114141f19af288 I0517 06:46:00.269109 17529 tablet_copy_source_session.cc:202] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:46:00.272310 17811 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0b8b163cec1348908d114141f19af288. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:00.285086 17811 ts_tablet_manager.cc:1173] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:46:00.285465 17811 tablet_copy_client.cc:565] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7: tablet copy: Starting download of 0 data blocks... I0517 06:46:00.285789 17811 tablet_copy_client.cc:528] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7: tablet copy: Starting download of 1 WAL segments... I0517 06:46:00.289961 17811 tablet_copy_client.cc:404] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:46:00.293833 17811 ts_tablet_manager.cc:1028] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7: Bootstrapping tablet I0517 06:46:00.294308 17811 tablet_bootstrap.cc:436] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7: Bootstrap starting. I0517 06:46:00.325112 17811 log.cc:520] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7: Log is configured to *not* fsync() on all Append() calls I0517 06:46:00.503612 17811 tablet_bootstrap.cc:436] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=127 overwritten=0 applied=127 ignored=0} inserts{seen=125 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0517 06:46:00.504356 17811 tablet_bootstrap.cc:436] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7: Bootstrap complete. I0517 06:46:00.504747 17811 ts_tablet_manager.cc:1045] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7: Time spent bootstrapping tablet: real 0.211s user 0.184s sys 0.024s I0517 06:46:00.512688 17811 raft_consensus.cc:303] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } peers { permanent_uuid: "4d7de5247d5c4bebb8312c873b9f10b7" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 35896 } } I0517 06:46:00.513250 17811 raft_consensus.cc:604] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7de5247d5c4bebb8312c873b9f10b7, State: Initialized, Role: FOLLOWER I0517 06:46:00.513715 17811 consensus_queue.cc:227] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 127, Last appended: 1.127, Last appended by leader: 127, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "5da82191918541a0ba8e299970b150a0" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 54414 } } peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } peers { permanent_uuid: "4d7de5247d5c4bebb8312c873b9f10b7" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 35896 } } I0517 06:46:00.516948 17811 ts_tablet_manager.cc:1073] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s I0517 06:46:00.518036 17529 tablet_copy_service.cc:323] P 68873422d063453797c080d8192b33be: Request end of tablet copy session 4d7de5247d5c4bebb8312c873b9f10b7-0b8b163cec1348908d114141f19af288 received from {username='slave'} at 127.4.96.5:39958 I0517 06:46:00.518265 17529 tablet_copy_service.cc:411] P 68873422d063453797c080d8192b33be: Ending tablet copy session 4d7de5247d5c4bebb8312c873b9f10b7-0b8b163cec1348908d114141f19af288 on tablet 0b8b163cec1348908d114141f19af288 with peer 4d7de5247d5c4bebb8312c873b9f10b7 I0517 06:46:00.879041 17633 raft_consensus.cc:1002] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.67->[1.68-1.190] Dedup: 1.127->[1.128-1.190] I0517 06:46:01.252771 17509 consensus_peers.cc:456] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be -> Peer 4d7de5247d5c4bebb8312c873b9f10b7 (127.4.96.5:35896): Closing peer: 4d7de5247d5c4bebb8312c873b9f10b7 I0517 06:46:01.252950 17509 consensus_peers.cc:456] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be -> Peer 7cd36755c08c4fa7bc42cb8b4d9be9cf (127.4.96.2:40420): Closing peer: 7cd36755c08c4fa7bc42cb8b4d9be9cf I0517 06:46:01.253155 17509 consensus_peers.cc:456] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be -> Peer 5da82191918541a0ba8e299970b150a0 (127.4.96.3:54414): Closing peer: 5da82191918541a0ba8e299970b150a0 I0517 06:46:01.253499 17509 consensus_queue.cc:204] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 1.214, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } peers { permanent_uuid: "4d7de5247d5c4bebb8312c873b9f10b7" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 35896 } } I0517 06:46:01.257122 17633 raft_consensus.cc:1057] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7 [term 1 FOLLOWER]: Refusing update from remote peer 68873422d063453797c080d8192b33be: Log matching property violated. Preceding OpId in replica: term: 1 index: 214. Preceding OpId from leader: term: 1 index: 215. (index mismatch) I0517 06:46:01.257225 17259 raft_consensus.cc:1057] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 1 FOLLOWER]: Refusing update from remote peer 68873422d063453797c080d8192b33be: Log matching property violated. Preceding OpId in replica: term: 1 index: 213. Preceding OpId from leader: term: 1 index: 215. (index mismatch) I0517 06:46:01.258165 17733 consensus_queue.cc:963] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d7de5247d5c4bebb8312c873b9f10b7" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 35896 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 213, Time since last communication: 0.000s I0517 06:46:01.258533 17774 consensus_queue.cc:963] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 213, Time since last communication: 0.000s I0517 06:46:01.264086 17259 raft_consensus.cc:2658] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf [term 1 FOLLOWER]: Committing config change with OpId 1.215: config changed from index 68 to 215, VOTER 5da82191918541a0ba8e299970b150a0 (127.4.96.3) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } peers { permanent_uuid: "4d7de5247d5c4bebb8312c873b9f10b7" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 35896 } } } I0517 06:46:01.264417 17733 raft_consensus.cc:2658] T 0b8b163cec1348908d114141f19af288 P 68873422d063453797c080d8192b33be [term 1 LEADER]: Committing config change with OpId 1.215: config changed from index 68 to 215, VOTER 5da82191918541a0ba8e299970b150a0 (127.4.96.3) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } peers { permanent_uuid: "4d7de5247d5c4bebb8312c873b9f10b7" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 35896 } } } I0517 06:46:01.264891 17633 raft_consensus.cc:2658] T 0b8b163cec1348908d114141f19af288 P 4d7de5247d5c4bebb8312c873b9f10b7 [term 1 FOLLOWER]: Committing config change with OpId 1.215: config changed from index 68 to 215, VOTER 5da82191918541a0ba8e299970b150a0 (127.4.96.3) evicted. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } peers { permanent_uuid: "4d7de5247d5c4bebb8312c873b9f10b7" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 35896 } } } I0517 06:46:01.272047 17011 catalog_manager.cc:3686] T 0b8b163cec1348908d114141f19af288 P 7cd36755c08c4fa7bc42cb8b4d9be9cf reported cstate change: config changed from index 68 to 215, VOTER 5da82191918541a0ba8e299970b150a0 (127.4.96.3) evicted. New cstate: current_term: 1 leader_uuid: "68873422d063453797c080d8192b33be" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "68873422d063453797c080d8192b33be" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 47104 } } peers { permanent_uuid: "7cd36755c08c4fa7bc42cb8b4d9be9cf" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 40420 } } peers { permanent_uuid: "4d7de5247d5c4bebb8312c873b9f10b7" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 35896 } } } I0517 06:46:01.304801 17011 catalog_manager.cc:3013] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 0b8b163cec1348908d114141f19af288 on 5da82191918541a0ba8e299970b150a0 (127.4.96.3:54414) (TS 5da82191918541a0ba8e299970b150a0 not found in new config with opid_index 215) I0517 06:46:01.308574 17365 tablet_service.cc:800] Processing DeleteTablet for tablet 0b8b163cec1348908d114141f19af288 with delete_type TABLET_DATA_TOMBSTONED (TS 5da82191918541a0ba8e299970b150a0 not found in new config with opid_index 215) from {username='slave'} at 127.0.0.1:60522 I0517 06:46:01.310369 17868 tablet_replica.cc:261] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: stopping tablet replica I0517 06:46:01.310854 17868 maintenance_manager.cc:237] P 5da82191918541a0ba8e299970b150a0: Unregistered op CompactRowSetsOp(0b8b163cec1348908d114141f19af288) I0517 06:46:01.311089 17868 maintenance_manager.cc:237] P 5da82191918541a0ba8e299970b150a0: Unregistered op MinorDeltaCompactionOp(0b8b163cec1348908d114141f19af288) I0517 06:46:01.311321 17868 maintenance_manager.cc:237] P 5da82191918541a0ba8e299970b150a0: Unregistered op MajorDeltaCompactionOp(0b8b163cec1348908d114141f19af288) I0517 06:46:01.311523 17868 maintenance_manager.cc:237] P 5da82191918541a0ba8e299970b150a0: Unregistered op UndoDeltaBlockGCOp(0b8b163cec1348908d114141f19af288) I0517 06:46:01.311759 17868 maintenance_manager.cc:237] P 5da82191918541a0ba8e299970b150a0: Unregistered op FlushMRSOp(0b8b163cec1348908d114141f19af288) I0517 06:46:01.311938 17868 maintenance_manager.cc:237] P 5da82191918541a0ba8e299970b150a0: Unregistered op FlushDeltaMemStoresOp(0b8b163cec1348908d114141f19af288) I0517 06:46:01.312103 17868 maintenance_manager.cc:237] P 5da82191918541a0ba8e299970b150a0: Unregistered op LogGCOp(0b8b163cec1348908d114141f19af288) I0517 06:46:01.312270 17868 raft_consensus.cc:2016] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [term 1 FOLLOWER]: Raft consensus shutting down. I0517 06:46:01.312520 17868 pending_rounds.cc:63] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: Trying to abort 1 pending transactions. I0517 06:46:01.312642 17868 pending_rounds.cc:68] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: Aborting transaction as it isn't in flight: id { term: 1 index: 214 } timestamp: 6252706042886758400 op_type: WRITE_OP write_request { tablet_id: "0b8b163cec1348908d114141f19af288" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\260\273KH\016\010t&\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "4ee8cef12cd46fb9" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6252706042826797056 } request_id { client_id: "149c8c747cf04fd3b65277b9bbd90537" seq_no: 211 first_incomplete_seq_no: 211 attempt_no: 0 } I0517 06:46:01.313652 17868 raft_consensus.cc:2043] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0 [term 1 FOLLOWER]: Raft consensus is shut down! I0517 06:46:01.316087 17868 ts_tablet_manager.cc:1369] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0517 06:46:01.324182 17868 ts_tablet_manager.cc:1382] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.214 I0517 06:46:01.324479 17868 log.cc:981] T 0b8b163cec1348908d114141f19af288 P 5da82191918541a0ba8e299970b150a0: Deleting WAL directory at /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/wals/0b8b163cec1348908d114141f19af288 I0517 06:46:01.325867 16995 catalog_manager.cc:2991] TS 5da82191918541a0ba8e299970b150a0 (127.4.96.3:54414): tablet 0b8b163cec1348908d114141f19af288 (table TestTable [id=71121150ba2b48b88f708e1ae44990cc]) successfully deleted I0517 06:46:02.362282 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 17182 W0517 06:46:02.390135 16993 connection.cc:512] client connection to 127.4.96.2:40420 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:40420): Connection reset by peer (error 104) W0517 06:46:02.390240 16984 connection.cc:512] client connection to 127.4.96.2:40420 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:40420): Connection reset by peer (error 104) W0517 06:46:02.391177 17439 connection.cc:512] client connection to 127.4.96.2:40420 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:40420): Connection reset by peer (error 104) W0517 06:46:02.392057 17315 connection.cc:512] server connection from 127.4.96.2:60769 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:60769): Connection reset by peer (error 104) I0517 06:46:02.392199 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 17307 W0517 06:46:02.393540 17439 connection.cc:512] server connection from 127.4.96.2:53613 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:53613): Connection reset by peer (error 104) W0517 06:46:02.402400 16994 connection.cc:512] server connection from 127.4.96.2:53271 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:53271): Connection reset by peer (error 104) W0517 06:46:02.430052 16995 connection.cc:512] client connection to 127.4.96.3:54414 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:54414): Connection reset by peer (error 104) W0517 06:46:02.430078 17441 connection.cc:512] client connection to 127.4.96.3:54414 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:54414): Connection reset by peer (error 104) W0517 06:46:02.430097 16984 connection.cc:512] client connection to 127.4.96.3:54414 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:54414): Connection reset by peer (error 104) W0517 06:46:02.430865 16995 connection.cc:512] server connection from 127.4.96.3:35893 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:35893): Connection reset by peer (error 104) I0517 06:46:02.431119 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 17431 W0517 06:46:02.470129 16993 connection.cc:512] server connection from 127.4.96.4:44712 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:44712): Connection reset by peer (error 104) W0517 06:46:02.470139 16984 connection.cc:512] client connection to 127.4.96.4:47104 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:47104): Connection reset by peer (error 104) W0517 06:46:02.470721 16997 connection.cc:512] client connection to 127.4.96.4:47104 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:47104): Connection reset by peer (error 104) I0517 06:46:02.471009 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 17555 W0517 06:46:02.497941 16994 connection.cc:512] server connection from 127.4.96.5:56815 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:56815): Connection reset by peer (error 104) W0517 06:46:02.497946 16984 connection.cc:512] client connection to 127.4.96.5:35896 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:35896): Connection reset by peer (error 104) W0517 06:46:02.498894 16981 connection.cc:512] client connection to 127.4.96.5:35896 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:35896): Connection reset by peer (error 104) I0517 06:46:02.499475 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 16985 W0517 06:46:02.529106 16984 connection.cc:512] client connection to 127.4.96.62:36482 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:36482): Connection reset by peer (error 104) [ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/2 (7246 ms) [ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/3 I0517 06:46:02.543406 4480 test_util.cc:200] Using random seed: 811362584 I0517 06:46:02.546375 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:46:02.546483 4480 ts_itest-base.cc:114] -------------- I0517 06:46:02.546587 4480 ts_itest-base.cc:115] 5 tablet servers I0517 06:46:02.546672 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:46:02.546759 4480 ts_itest-base.cc:117] -------------- I0517 06:46:02.548794 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:46703 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:02.663528 17876 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:02.663874 17876 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:02.672860 17876 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:02.673032 17876 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:46:02.673146 17876 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:02.673260 17876 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:46:02.673391 17876 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:46:02.687256 17876 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:46703 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:02.694933 17876 master_main.cc:80] Initializing master server... I0517 06:46:02.696436 17876 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 180319us I0517 06:46:02.697183 17876 fs_manager.cc:260] Metadata directory not provided I0517 06:46:02.697321 17876 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:02.697537 17876 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:46:02.697656 17876 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:02.704887 17876 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "f764d804a93e440ba2cc8161931cf324" format_stamp: "Formatted at 2018-05-17 06:46:02 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:02.705559 17876 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "f764d804a93e440ba2cc8161931cf324" format_stamp: "Formatted at 2018-05-17 06:46:02 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:02.715013 17876 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.008s I0517 06:46:02.716138 17876 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:02.716545 17876 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:02.722884 17876 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:46:02.723000 17876 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "f764d804a93e440ba2cc8161931cf324" format_stamp: "Formatted at 2018-05-17 06:46:02 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:02.723208 17876 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:02.931653 17876 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:02.933509 17876 master_main.cc:83] Starting Master server... I0517 06:46:02.972637 17876 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:46703 I0517 06:46:02.973501 17876 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:46:02.973609 17876 webserver.cc:184] Document root disabled I0517 06:46:02.974807 17876 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:56359/ I0517 06:46:02.975530 17876 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:46:02.978106 17937 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:02.983651 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 17876 I0517 06:46:02.984570 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:46703 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:02.991618 17937 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324: Bootstrap starting. I0517 06:46:02.993799 17937 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324: No blocks or log segments found. Creating new log. I0517 06:46:02.994935 17937 log.cc:520] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324: Log is configured to *not* fsync() on all Append() calls I0517 06:46:02.997244 17937 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324: No bootstrap required, opened a new log I0517 06:46:03.008836 17937 raft_consensus.cc:303] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f764d804a93e440ba2cc8161931cf324" member_type: VOTER } I0517 06:46:03.009137 17937 raft_consensus.cc:329] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:46:03.009271 17937 raft_consensus.cc:604] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f764d804a93e440ba2cc8161931cf324, State: Initialized, Role: FOLLOWER I0517 06:46:03.009719 17937 consensus_queue.cc:227] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f764d804a93e440ba2cc8161931cf324" member_type: VOTER } I0517 06:46:03.010076 17937 raft_consensus.cc:343] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:46:03.010213 17937 raft_consensus.cc:435] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:46:03.010356 17937 raft_consensus.cc:2704] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324: Snoozing failure detection for 1.708s (starting election) I0517 06:46:03.010499 17937 raft_consensus.cc:2756] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:46:03.012900 17937 raft_consensus.cc:457] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f764d804a93e440ba2cc8161931cf324" member_type: VOTER } I0517 06:46:03.013309 17937 leader_election.cc:270] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:46:03.014896 17940 raft_consensus.cc:2704] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324: Snoozing failure detection for 1.788s (election complete) I0517 06:46:03.015197 17940 raft_consensus.cc:2511] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:46:03.026403 17940 raft_consensus.cc:571] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [term 1 LEADER]: Becoming Leader. State: Replica: f764d804a93e440ba2cc8161931cf324, State: Running, Role: LEADER I0517 06:46:03.026804 17940 consensus_queue.cc:204] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f764d804a93e440ba2cc8161931cf324" member_type: VOTER } I0517 06:46:03.030165 17937 sys_catalog.cc:335] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "f764d804a93e440ba2cc8161931cf324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f764d804a93e440ba2cc8161931cf324" member_type: VOTER } } I0517 06:46:03.030616 17937 sys_catalog.cc:338] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [sys.catalog]: This master's current role is: LEADER I0517 06:46:03.030642 17941 sys_catalog.cc:335] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f764d804a93e440ba2cc8161931cf324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f764d804a93e440ba2cc8161931cf324" member_type: VOTER } } I0517 06:46:03.091586 17941 sys_catalog.cc:338] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [sys.catalog]: This master's current role is: LEADER I0517 06:46:03.030935 17942 sys_catalog.cc:335] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f764d804a93e440ba2cc8161931cf324. Latest consensus state: current_term: 1 leader_uuid: "f764d804a93e440ba2cc8161931cf324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f764d804a93e440ba2cc8161931cf324" member_type: VOTER } } I0517 06:46:03.093647 17942 sys_catalog.cc:338] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [sys.catalog]: This master's current role is: LEADER I0517 06:46:03.094872 17937 sys_catalog.cc:422] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:46:03.095157 17944 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:46:03.096494 17876 master_main.cc:86] Master server successfully started. I0517 06:46:03.099932 17944 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:46:03.123440 17944 catalog_manager.cc:899] Generated new certificate authority record WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:03.124666 17938 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true I0517 06:46:03.124902 17944 catalog_manager.cc:1021] Loading token signing keys... W0517 06:46:03.125141 17938 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:46:03.135135 17944 catalog_manager.cc:3962] T 00000000000000000000000000000000 P f764d804a93e440ba2cc8161931cf324: Generated new TSK 0 W0517 06:46:03.139701 17938 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:03.139940 17938 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:03.140429 17938 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:46:03.160802 17938 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:46703 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:03.173982 17938 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:03.176429 17938 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 180819us I0517 06:46:03.177049 17938 fs_manager.cc:260] Metadata directory not provided I0517 06:46:03.177220 17938 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:03.177489 17938 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:46:03.177652 17938 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:03.190410 17938 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "976270186b154e39ac3c1d31e10e4988" format_stamp: "Formatted at 2018-05-17 06:46:03 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:03.191357 17938 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "976270186b154e39ac3c1d31e10e4988" format_stamp: "Formatted at 2018-05-17 06:46:03 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:03.205471 17938 fs_manager.cc:495] Time spent creating directory manager: real 0.014s user 0.012s sys 0.004s I0517 06:46:03.207185 17938 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:03.207716 17938 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:03.218181 17938 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:46:03.218331 17938 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "976270186b154e39ac3c1d31e10e4988" format_stamp: "Formatted at 2018-05-17 06:46:03 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:03.218602 17938 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:03.396739 17938 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:03.399873 17938 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:03.400993 17938 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:03.936570 17938 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:55449 I0517 06:46:03.937906 17938 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:46:03.938128 17938 webserver.cc:184] Document root disabled I0517 06:46:03.939800 17938 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:38221/ I0517 06:46:03.940667 17938 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:46:03.945984 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 17938 I0517 06:46:03.947198 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:46703 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:03.974128 18069 heartbeater.cc:346] Connected to a master server at 127.4.96.62:46703 I0517 06:46:03.974436 18069 heartbeater.cc:426] Registering TS with master... I0517 06:46:03.975090 18069 heartbeater.cc:475] Master 127.4.96.62:46703 requested a full tablet report, sending... I0517 06:46:03.976891 17938 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:03.977190 17904 ts_manager.cc:81] Registered new tserver with Master: 976270186b154e39ac3c1d31e10e4988 (127.4.96.1:55449) I0517 06:46:03.979737 17904 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:36348 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:04.073081 18073 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:04.073407 18073 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:04.081284 18073 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:04.081446 18073 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:04.081738 18073 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:46:04.094624 18073 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:46703 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:04.102702 18073 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:04.104450 18073 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 181319us I0517 06:46:04.104858 18073 fs_manager.cc:260] Metadata directory not provided I0517 06:46:04.104967 18073 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:04.105149 18073 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:46:04.105252 18073 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:04.112830 18073 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "bdb84e4dc496433f957e3188ed2d22dc" format_stamp: "Formatted at 2018-05-17 06:46:04 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:04.113476 18073 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "bdb84e4dc496433f957e3188ed2d22dc" format_stamp: "Formatted at 2018-05-17 06:46:04 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:04.123363 18073 fs_manager.cc:495] Time spent creating directory manager: real 0.010s user 0.008s sys 0.004s I0517 06:46:04.124518 18073 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:04.124903 18073 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:04.131418 18073 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0517 06:46:04.131537 18073 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "bdb84e4dc496433f957e3188ed2d22dc" format_stamp: "Formatted at 2018-05-17 06:46:04 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:04.131760 18073 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:04.342593 18073 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:04.345119 18073 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:04.346115 18073 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:04.432726 18073 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:39828 I0517 06:46:04.433596 18073 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:46:04.433703 18073 webserver.cc:184] Document root disabled I0517 06:46:04.434988 18073 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:58874/ I0517 06:46:04.435665 18073 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:46:04.439095 18073 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:04.445199 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 18073 I0517 06:46:04.446197 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:46703 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:04.456246 18194 heartbeater.cc:346] Connected to a master server at 127.4.96.62:46703 I0517 06:46:04.456501 18194 heartbeater.cc:426] Registering TS with master... I0517 06:46:04.456970 18194 heartbeater.cc:475] Master 127.4.96.62:46703 requested a full tablet report, sending... I0517 06:46:04.458312 17904 ts_manager.cc:81] Registered new tserver with Master: bdb84e4dc496433f957e3188ed2d22dc (127.4.96.2:39828) I0517 06:46:04.459463 17904 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:59716 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:04.550563 18197 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:04.550882 18197 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:04.558714 18197 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:04.558867 18197 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:04.559154 18197 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:46:04.571228 18197 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:46703 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:04.579282 18197 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:04.581019 18197 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 181319us I0517 06:46:04.581439 18197 fs_manager.cc:260] Metadata directory not provided I0517 06:46:04.581550 18197 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:04.581737 18197 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:46:04.581845 18197 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:04.589304 18197 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "c72d9ae7bc4d4d3b8e61e4d35a37b843" format_stamp: "Formatted at 2018-05-17 06:46:04 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:04.589960 18197 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "c72d9ae7bc4d4d3b8e61e4d35a37b843" format_stamp: "Formatted at 2018-05-17 06:46:04 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:04.599238 18197 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:46:04.600342 18197 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:04.600718 18197 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:04.607312 18197 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0517 06:46:04.607429 18197 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "c72d9ae7bc4d4d3b8e61e4d35a37b843" format_stamp: "Formatted at 2018-05-17 06:46:04 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:04.607622 18197 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:04.761395 18197 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:04.763777 18197 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:04.764703 18197 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:04.849283 18197 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:59498 I0517 06:46:04.850165 18197 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:46:04.850263 18197 webserver.cc:184] Document root disabled I0517 06:46:04.851442 18197 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:53948/ I0517 06:46:04.852124 18197 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:46:04.854955 18197 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:04.862131 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 18197 I0517 06:46:04.863188 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:46703 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:04.872383 18319 heartbeater.cc:346] Connected to a master server at 127.4.96.62:46703 I0517 06:46:04.872622 18319 heartbeater.cc:426] Registering TS with master... I0517 06:46:04.873049 18319 heartbeater.cc:475] Master 127.4.96.62:46703 requested a full tablet report, sending... I0517 06:46:04.874354 17904 ts_manager.cc:81] Registered new tserver with Master: c72d9ae7bc4d4d3b8e61e4d35a37b843 (127.4.96.3:59498) I0517 06:46:04.875479 17904 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:45875 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:04.967756 18322 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:04.968089 18322 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:04.975900 18322 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:04.976074 18322 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:04.976379 18322 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:46:04.983036 18069 heartbeater.cc:467] Master 127.4.96.62:46703 was elected leader, sending a full tablet report... I0517 06:46:04.990156 18322 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:46703 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:04.998193 18322 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:04.999966 18322 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 181319us I0517 06:46:05.000399 18322 fs_manager.cc:260] Metadata directory not provided I0517 06:46:05.000531 18322 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:05.000739 18322 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:46:05.000857 18322 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:05.008350 18322 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "914aaa60c2844b42a89d7411a5c778b3" format_stamp: "Formatted at 2018-05-17 06:46:05 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:05.009035 18322 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "914aaa60c2844b42a89d7411a5c778b3" format_stamp: "Formatted at 2018-05-17 06:46:05 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:05.018319 18322 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:46:05.019439 18322 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:05.019836 18322 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:05.026155 18322 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0517 06:46:05.026271 18322 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "914aaa60c2844b42a89d7411a5c778b3" format_stamp: "Formatted at 2018-05-17 06:46:05 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:05.026471 18322 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:05.187618 18322 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:05.190034 18322 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:05.190946 18322 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:05.277173 18322 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:39320 I0517 06:46:05.278086 18322 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:46:05.278194 18322 webserver.cc:184] Document root disabled I0517 06:46:05.279386 18322 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:58545/ I0517 06:46:05.280069 18322 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:46:05.282565 18322 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:05.290274 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 18322 I0517 06:46:05.291185 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.4.96.5:0 --local_ip_for_outbound_sockets=127.4.96.5 --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:46703 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:05.305063 18443 heartbeater.cc:346] Connected to a master server at 127.4.96.62:46703 I0517 06:46:05.305332 18443 heartbeater.cc:426] Registering TS with master... I0517 06:46:05.305821 18443 heartbeater.cc:475] Master 127.4.96.62:46703 requested a full tablet report, sending... I0517 06:46:05.307142 17904 ts_manager.cc:81] Registered new tserver with Master: 914aaa60c2844b42a89d7411a5c778b3 (127.4.96.4:39320) I0517 06:46:05.308233 17904 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:42149 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:05.396539 18446 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:05.396859 18446 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:05.404636 18446 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:05.404790 18446 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:05.405083 18446 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.5 I0517 06:46:05.417464 18446 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:46703 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.5 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:05.425423 18446 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:05.427165 18446 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 181819us I0517 06:46:05.427608 18446 fs_manager.cc:260] Metadata directory not provided I0517 06:46:05.427755 18446 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:05.428014 18446 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0517 06:46:05.428184 18446 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:05.435660 18446 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "cb7678b41edd473396efaafe766253da" format_stamp: "Formatted at 2018-05-17 06:46:05 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:05.436319 18446 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "cb7678b41edd473396efaafe766253da" format_stamp: "Formatted at 2018-05-17 06:46:05 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:05.445618 18446 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s I0517 06:46:05.446728 18446 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:05.447124 18446 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:05.453341 18446 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:46:05.453447 18446 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal uuid: "cb7678b41edd473396efaafe766253da" format_stamp: "Formatted at 2018-05-17 06:46:05 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:05.453626 18446 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:05.461719 18194 heartbeater.cc:467] Master 127.4.96.62:46703 was elected leader, sending a full tablet report... I0517 06:46:05.646044 18446 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:05.648392 18446 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:05.649294 18446 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:05.733175 18446 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.5:59858 I0517 06:46:05.734104 18446 webserver.cc:173] Starting webserver on 127.4.96.5:0 I0517 06:46:05.734210 18446 webserver.cc:184] Document root disabled I0517 06:46:05.735450 18446 webserver.cc:311] Webserver started. Bound to: http://127.4.96.5:45093/ I0517 06:46:05.736129 18446 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb I0517 06:46:05.736680 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 18446 I0517 06:46:05.743371 18446 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:05.760979 18567 heartbeater.cc:346] Connected to a master server at 127.4.96.62:46703 I0517 06:46:05.761299 18567 heartbeater.cc:426] Registering TS with master... I0517 06:46:05.761821 18567 heartbeater.cc:475] Master 127.4.96.62:46703 requested a full tablet report, sending... I0517 06:46:05.763099 17904 ts_manager.cc:81] Registered new tserver with Master: cb7678b41edd473396efaafe766253da (127.4.96.5:59858) I0517 06:46:05.764437 17904 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.5:46035 I0517 06:46:05.765194 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:46:05.782843 17904 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45916: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:46:05.830241 18131 tablet_service.cc:759] Processing CreateTablet for tablet e9da75b6930d48cebd69f0413be12078 (table=TestTable [id=32ce2928a539415e9f8388736c762ee1]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:46:05.831141 18131 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e9da75b6930d48cebd69f0413be12078. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:05.831619 18006 tablet_service.cc:759] Processing CreateTablet for tablet e9da75b6930d48cebd69f0413be12078 (table=TestTable [id=32ce2928a539415e9f8388736c762ee1]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:46:05.832244 18006 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e9da75b6930d48cebd69f0413be12078. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:05.844126 18504 tablet_service.cc:759] Processing CreateTablet for tablet e9da75b6930d48cebd69f0413be12078 (table=TestTable [id=32ce2928a539415e9f8388736c762ee1]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:46:05.845249 18504 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e9da75b6930d48cebd69f0413be12078. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:05.845315 18006 ts_tablet_manager.cc:1173] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: Registered tablet (data state: TABLET_DATA_READY) I0517 06:46:05.846180 18131 ts_tablet_manager.cc:1173] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: Registered tablet (data state: TABLET_DATA_READY) I0517 06:46:05.862481 18504 ts_tablet_manager.cc:1173] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da: Registered tablet (data state: TABLET_DATA_READY) I0517 06:46:05.866257 18587 ts_tablet_manager.cc:1028] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: Bootstrapping tablet I0517 06:46:05.866469 18588 ts_tablet_manager.cc:1028] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: Bootstrapping tablet I0517 06:46:05.866704 18587 tablet_bootstrap.cc:436] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: Bootstrap starting. I0517 06:46:05.866966 18588 tablet_bootstrap.cc:436] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: Bootstrap starting. I0517 06:46:05.870153 18588 tablet_bootstrap.cc:587] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: No blocks or log segments found. Creating new log. I0517 06:46:05.870573 18587 tablet_bootstrap.cc:587] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: No blocks or log segments found. Creating new log. I0517 06:46:05.872115 18587 log.cc:520] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: Log is configured to *not* fsync() on all Append() calls I0517 06:46:05.873700 18589 ts_tablet_manager.cc:1028] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da: Bootstrapping tablet I0517 06:46:05.874166 18589 tablet_bootstrap.cc:436] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da: Bootstrap starting. I0517 06:46:05.874236 18588 log.cc:520] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: Log is configured to *not* fsync() on all Append() calls I0517 06:46:05.877120 18589 tablet_bootstrap.cc:587] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da: No blocks or log segments found. Creating new log. I0517 06:46:05.877331 18588 tablet_bootstrap.cc:436] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: No bootstrap required, opened a new log I0517 06:46:05.877673 18319 heartbeater.cc:467] Master 127.4.96.62:46703 was elected leader, sending a full tablet report... I0517 06:46:05.877705 18588 ts_tablet_manager.cc:1045] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s I0517 06:46:05.878657 18589 log.cc:520] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da: Log is configured to *not* fsync() on all Append() calls I0517 06:46:05.880599 18587 tablet_bootstrap.cc:436] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: No bootstrap required, opened a new log I0517 06:46:05.880861 18587 ts_tablet_manager.cc:1045] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s I0517 06:46:05.890524 18587 raft_consensus.cc:303] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } I0517 06:46:05.891054 18587 raft_consensus.cc:329] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:46:05.891185 18587 raft_consensus.cc:604] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 976270186b154e39ac3c1d31e10e4988, State: Initialized, Role: FOLLOWER I0517 06:46:05.891779 18589 tablet_bootstrap.cc:436] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da: No bootstrap required, opened a new log I0517 06:46:05.891558 18587 consensus_queue.cc:227] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } I0517 06:46:05.892235 18589 ts_tablet_manager.cc:1045] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s I0517 06:46:05.893563 18588 raft_consensus.cc:303] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } I0517 06:46:05.894348 18588 raft_consensus.cc:329] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:46:05.894562 18588 raft_consensus.cc:604] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb84e4dc496433f957e3188ed2d22dc, State: Initialized, Role: FOLLOWER I0517 06:46:05.895095 18588 consensus_queue.cc:227] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } I0517 06:46:05.898772 18587 ts_tablet_manager.cc:1073] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: Time spent starting tablet: real 0.018s user 0.012s sys 0.008s I0517 06:46:05.903311 18588 ts_tablet_manager.cc:1073] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: Time spent starting tablet: real 0.025s user 0.016s sys 0.004s I0517 06:46:05.906076 18589 raft_consensus.cc:303] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } I0517 06:46:05.906584 18589 raft_consensus.cc:329] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:46:05.906744 18589 raft_consensus.cc:604] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb7678b41edd473396efaafe766253da, State: Initialized, Role: FOLLOWER I0517 06:46:05.907110 18589 consensus_queue.cc:227] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } I0517 06:46:05.908711 18567 heartbeater.cc:467] Master 127.4.96.62:46703 was elected leader, sending a full tablet report... I0517 06:46:05.909379 18589 ts_tablet_manager.cc:1073] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s W0517 06:46:05.978358 18070 tablet.cc:1678] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:46:05.940129 18195 tablet.cc:1678] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:46:06.020874 18594 raft_consensus.cc:435] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:46:06.021185 18594 raft_consensus.cc:2704] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: Snoozing failure detection for 2.247s (starting election) I0517 06:46:06.021450 18594 raft_consensus.cc:457] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } I0517 06:46:06.022369 18594 leader_election.cc:243] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 976270186b154e39ac3c1d31e10e4988 (127.4.96.1:55449) I0517 06:46:06.022804 18594 leader_election.cc:243] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer cb7678b41edd473396efaafe766253da (127.4.96.5:59858) I0517 06:46:06.041224 18026 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e9da75b6930d48cebd69f0413be12078" candidate_uuid: "bdb84e4dc496433f957e3188ed2d22dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "976270186b154e39ac3c1d31e10e4988" is_pre_election: true I0517 06:46:06.041712 18026 raft_consensus.cc:2704] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: Snoozing failure detection for 2.053s (vote granted) I0517 06:46:06.041891 18026 raft_consensus.cc:2219] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdb84e4dc496433f957e3188ed2d22dc in term 0. I0517 06:46:06.042934 18085 leader_election.cc:379] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [CANDIDATE]: Term 1 pre-election: Vote granted by peer 976270186b154e39ac3c1d31e10e4988 (127.4.96.1:55449) I0517 06:46:06.043170 18085 leader_election.cc:270] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:46:06.043661 18594 raft_consensus.cc:2704] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: Snoozing failure detection for 2.159s (election complete) I0517 06:46:06.043905 18594 raft_consensus.cc:2511] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:46:06.044096 18594 raft_consensus.cc:435] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:46:06.044263 18594 raft_consensus.cc:2704] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: Snoozing failure detection for 1.814s (starting election) I0517 06:46:06.044582 18594 raft_consensus.cc:2756] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 0 FOLLOWER]: Advancing to term 1 I0517 06:46:06.049118 18594 raft_consensus.cc:457] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } I0517 06:46:06.050089 18594 leader_election.cc:243] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [CANDIDATE]: Term 1 election: Requesting vote from peer 976270186b154e39ac3c1d31e10e4988 (127.4.96.1:55449) I0517 06:46:06.050477 18594 leader_election.cc:243] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [CANDIDATE]: Term 1 election: Requesting vote from peer cb7678b41edd473396efaafe766253da (127.4.96.5:59858) I0517 06:46:06.051229 18524 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e9da75b6930d48cebd69f0413be12078" candidate_uuid: "bdb84e4dc496433f957e3188ed2d22dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb7678b41edd473396efaafe766253da" is_pre_election: true I0517 06:46:06.052646 18026 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e9da75b6930d48cebd69f0413be12078" candidate_uuid: "bdb84e4dc496433f957e3188ed2d22dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "976270186b154e39ac3c1d31e10e4988" I0517 06:46:06.053030 18026 raft_consensus.cc:2756] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:46:06.053231 18026 raft_consensus.cc:2704] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: Snoozing failure detection for 1.810s (vote granted) I0517 06:46:06.055976 18026 raft_consensus.cc:2219] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdb84e4dc496433f957e3188ed2d22dc in term 1. W0517 06:46:05.993381 18568 tablet.cc:1678] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:46:06.056682 18085 leader_election.cc:379] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [CANDIDATE]: Term 1 election: Vote granted by peer 976270186b154e39ac3c1d31e10e4988 (127.4.96.1:55449) I0517 06:46:06.056849 18085 leader_election.cc:270] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:46:06.057209 18594 raft_consensus.cc:2704] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: Snoozing failure detection for 2.080s (election complete) I0517 06:46:06.057379 18594 raft_consensus.cc:2511] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:46:06.068274 18594 raft_consensus.cc:571] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 1 LEADER]: Becoming Leader. State: Replica: bdb84e4dc496433f957e3188ed2d22dc, State: Running, Role: LEADER I0517 06:46:06.068676 18594 consensus_queue.cc:204] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } I0517 06:46:06.077239 17904 catalog_manager.cc:3686] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc reported cstate change: term changed from 0 to 1, leader changed from to bdb84e4dc496433f957e3188ed2d22dc (127.4.96.2). New cstate: current_term: 1 leader_uuid: "bdb84e4dc496433f957e3188ed2d22dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } health_report { overall_health: UNKNOWN } } } I0517 06:46:06.002136 18595 raft_consensus.cc:435] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:46:06.080054 18595 raft_consensus.cc:2704] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da: Snoozing failure detection for 1.779s (starting election) I0517 06:46:06.080349 18595 raft_consensus.cc:457] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } I0517 06:46:06.081360 18595 leader_election.cc:243] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer bdb84e4dc496433f957e3188ed2d22dc (127.4.96.2:39828) I0517 06:46:06.083091 18524 raft_consensus.cc:2704] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da: Snoozing failure detection for 2.035s (vote granted) I0517 06:46:06.083351 18524 raft_consensus.cc:2219] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdb84e4dc496433f957e3188ed2d22dc in term 0. I0517 06:46:06.084054 18595 leader_election.cc:243] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 976270186b154e39ac3c1d31e10e4988 (127.4.96.1:55449) I0517 06:46:06.088457 18085 leader_election.cc:379] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [CANDIDATE]: Term 1 pre-election: Vote granted by peer cb7678b41edd473396efaafe766253da (127.4.96.5:59858) I0517 06:46:06.052752 18523 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e9da75b6930d48cebd69f0413be12078" candidate_uuid: "bdb84e4dc496433f957e3188ed2d22dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb7678b41edd473396efaafe766253da" I0517 06:46:06.090435 18523 raft_consensus.cc:2756] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 0 FOLLOWER]: Advancing to term 1 I0517 06:46:06.090662 18523 raft_consensus.cc:2704] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da: Snoozing failure detection for 1.567s (vote granted) I0517 06:46:06.091586 18151 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e9da75b6930d48cebd69f0413be12078" candidate_uuid: "cb7678b41edd473396efaafe766253da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb84e4dc496433f957e3188ed2d22dc" is_pre_election: true I0517 06:46:06.092231 18151 raft_consensus.cc:2180] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate cb7678b41edd473396efaafe766253da for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:46:06.093495 18456 leader_election.cc:393] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [CANDIDATE]: Term 1 pre-election: Vote denied by peer bdb84e4dc496433f957e3188ed2d22dc (127.4.96.2:39828). Message: Invalid argument: T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate cb7678b41edd473396efaafe766253da for term 1 because replica is either leader or believes a valid leader to be alive. I0517 06:46:06.094584 18523 raft_consensus.cc:2219] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdb84e4dc496433f957e3188ed2d22dc in term 1. I0517 06:46:06.095544 18085 leader_election.cc:379] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [CANDIDATE]: Term 1 election: Vote granted by peer cb7678b41edd473396efaafe766253da (127.4.96.5:59858) I0517 06:46:06.098851 18026 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e9da75b6930d48cebd69f0413be12078" candidate_uuid: "cb7678b41edd473396efaafe766253da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "976270186b154e39ac3c1d31e10e4988" is_pre_election: true I0517 06:46:06.099270 18026 raft_consensus.cc:2149] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cb7678b41edd473396efaafe766253da in current term 1: Already voted for candidate bdb84e4dc496433f957e3188ed2d22dc in this term. I0517 06:46:06.100037 18458 leader_election.cc:393] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [CANDIDATE]: Term 1 pre-election: Vote denied by peer 976270186b154e39ac3c1d31e10e4988 (127.4.96.1:55449). Message: Invalid argument: T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cb7678b41edd473396efaafe766253da in current term 1: Already voted for candidate bdb84e4dc496433f957e3188ed2d22dc in this term. I0517 06:46:06.100224 18458 leader_election.cc:270] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0517 06:46:06.100600 18595 raft_consensus.cc:2704] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da: Snoozing failure detection for 2.247s (election complete) I0517 06:46:06.100755 18595 raft_consensus.cc:2459] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0517 06:46:06.112501 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:46:06.114497 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 976270186b154e39ac3c1d31e10e4988 to finish bootstrapping I0517 06:46:06.123201 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bdb84e4dc496433f957e3188ed2d22dc to finish bootstrapping I0517 06:46:06.131461 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c72d9ae7bc4d4d3b8e61e4d35a37b843 to finish bootstrapping I0517 06:46:06.142006 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 914aaa60c2844b42a89d7411a5c778b3 to finish bootstrapping I0517 06:46:06.151921 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver cb7678b41edd473396efaafe766253da to finish bootstrapping I0517 06:46:06.160406 4480 test_util.cc:200] Using random seed: 814979588 I0517 06:46:06.176511 4480 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0517 06:46:06.221973 18523 raft_consensus.cc:1057] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 1 FOLLOWER]: Refusing update from remote peer bdb84e4dc496433f957e3188ed2d22dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0517 06:46:06.222326 18026 raft_consensus.cc:1057] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 1 FOLLOWER]: Refusing update from remote peer bdb84e4dc496433f957e3188ed2d22dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0517 06:46:06.223675 18598 consensus_queue.cc:963] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:46:06.224977 18598 consensus_queue.cc:963] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:46:06.310449 18443 heartbeater.cc:467] Master 127.4.96.62:46703 was elected leader, sending a full tablet report... W0517 06:46:06.340833 18609 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:46:06.844571 18151 consensus_peers.cc:456] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc -> Peer cb7678b41edd473396efaafe766253da (127.4.96.5:59858): Closing peer: cb7678b41edd473396efaafe766253da I0517 06:46:06.844776 18151 consensus_peers.cc:456] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc -> Peer 976270186b154e39ac3c1d31e10e4988 (127.4.96.1:55449): Closing peer: 976270186b154e39ac3c1d31e10e4988 I0517 06:46:06.845058 18151 consensus_queue.cc:204] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 1.51, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: true } } I0517 06:46:06.850682 18524 raft_consensus.cc:1057] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 1 FOLLOWER]: Refusing update from remote peer bdb84e4dc496433f957e3188ed2d22dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 51. Preceding OpId from leader: term: 1 index: 52. (index mismatch) I0517 06:46:06.851513 18025 raft_consensus.cc:1057] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 1 FOLLOWER]: Refusing update from remote peer bdb84e4dc496433f957e3188ed2d22dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 51. Preceding OpId from leader: term: 1 index: 52. (index mismatch) I0517 06:46:06.852695 18598 consensus_queue.cc:963] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 50, Time since last communication: 0.000s I0517 06:46:06.853238 18594 consensus_queue.cc:963] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s I0517 06:46:06.858659 18594 raft_consensus.cc:2658] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 1 LEADER]: Committing config change with OpId 1.52: config changed from index -1 to 52, NON_VOTER 914aaa60c2844b42a89d7411a5c778b3 (127.4.96.4) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: true } } } I0517 06:46:06.859452 18025 raft_consensus.cc:2658] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 1 FOLLOWER]: Committing config change with OpId 1.52: config changed from index -1 to 52, NON_VOTER 914aaa60c2844b42a89d7411a5c778b3 (127.4.96.4) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: true } } } W0517 06:46:06.860877 18085 consensus_peers.cc:434] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc -> Peer 914aaa60c2844b42a89d7411a5c778b3 (127.4.96.4:39320): Couldn't send request to peer 914aaa60c2844b42a89d7411a5c778b3 for tablet e9da75b6930d48cebd69f0413be12078. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9da75b6930d48cebd69f0413be12078. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:46:06.865061 18524 raft_consensus.cc:2658] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 1 FOLLOWER]: Committing config change with OpId 1.52: config changed from index -1 to 52, NON_VOTER 914aaa60c2844b42a89d7411a5c778b3 (127.4.96.4) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: true } } } I0517 06:46:06.867686 17903 catalog_manager.cc:3686] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 reported cstate change: config changed from index -1 to 52, NON_VOTER 914aaa60c2844b42a89d7411a5c778b3 (127.4.96.4) added. New cstate: current_term: 1 leader_uuid: "bdb84e4dc496433f957e3188ed2d22dc" committed_config { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: true } } } I0517 06:46:07.331861 18721 ts_tablet_manager.cc:669] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3: Initiating tablet copy from peer bdb84e4dc496433f957e3188ed2d22dc (127.4.96.2:39828) I0517 06:46:07.332293 18721 tablet_copy_client.cc:239] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.2:39828 I0517 06:46:07.347331 18171 tablet_copy_service.cc:133] P bdb84e4dc496433f957e3188ed2d22dc: Received BeginTabletCopySession request for tablet e9da75b6930d48cebd69f0413be12078 from peer 914aaa60c2844b42a89d7411a5c778b3 ({username='slave'} at 127.4.96.4:53845) I0517 06:46:07.347666 18171 tablet_copy_service.cc:154] P bdb84e4dc496433f957e3188ed2d22dc: Beginning new tablet copy session on tablet e9da75b6930d48cebd69f0413be12078 from peer 914aaa60c2844b42a89d7411a5c778b3 at {username='slave'} at 127.4.96.4:53845: session id = 914aaa60c2844b42a89d7411a5c778b3-e9da75b6930d48cebd69f0413be12078 I0517 06:46:07.354112 18171 tablet_copy_source_session.cc:202] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:46:07.358956 18721 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e9da75b6930d48cebd69f0413be12078. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:07.379099 18721 ts_tablet_manager.cc:1173] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:46:07.379947 18721 tablet_copy_client.cc:565] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3: tablet copy: Starting download of 0 data blocks... I0517 06:46:07.380295 18721 tablet_copy_client.cc:528] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3: tablet copy: Starting download of 1 WAL segments... I0517 06:46:07.384411 18721 tablet_copy_client.cc:404] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:46:07.388725 18721 ts_tablet_manager.cc:1028] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3: Bootstrapping tablet I0517 06:46:07.389240 18721 tablet_bootstrap.cc:436] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3: Bootstrap starting. I0517 06:46:07.419602 18721 log.cc:520] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3: Log is configured to *not* fsync() on all Append() calls I0517 06:46:07.589963 18721 tablet_bootstrap.cc:436] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=100 overwritten=0 applied=99 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0517 06:46:07.590837 18721 tablet_bootstrap.cc:436] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3: Bootstrap complete. I0517 06:46:07.591363 18721 ts_tablet_manager.cc:1045] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3: Time spent bootstrapping tablet: real 0.202s user 0.176s sys 0.028s I0517 06:46:07.601824 18721 raft_consensus.cc:303] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3 [term 1 LEARNER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: true } } I0517 06:46:07.607575 18721 raft_consensus.cc:604] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 914aaa60c2844b42a89d7411a5c778b3, State: Initialized, Role: LEARNER I0517 06:46:07.608060 18721 consensus_queue.cc:227] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 99, Last appended: 1.100, Last appended by leader: 100, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: true } } I0517 06:46:07.610612 18721 ts_tablet_manager.cc:1073] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3: Time spent starting tablet: real 0.019s user 0.012s sys 0.000s I0517 06:46:07.611605 18171 tablet_copy_service.cc:323] P bdb84e4dc496433f957e3188ed2d22dc: Request end of tablet copy session 914aaa60c2844b42a89d7411a5c778b3-e9da75b6930d48cebd69f0413be12078 received from {username='slave'} at 127.4.96.4:53845 I0517 06:46:07.611825 18171 tablet_copy_service.cc:411] P bdb84e4dc496433f957e3188ed2d22dc: Ending tablet copy session 914aaa60c2844b42a89d7411a5c778b3-e9da75b6930d48cebd69f0413be12078 on tablet e9da75b6930d48cebd69f0413be12078 with peer 914aaa60c2844b42a89d7411a5c778b3 W0517 06:46:07.722664 17889 debug-util.cc:397] Leaking SignalData structure 0x7b080002b080 after lost signal to thread 17877 I0517 06:46:07.923696 18400 raft_consensus.cc:1002] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.51->[1.52-1.140] Dedup: 1.100->[1.101-1.140] I0517 06:46:08.049860 18675 raft_consensus.cc:866] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc: attempting to promote NON_VOTER 914aaa60c2844b42a89d7411a5c778b3 to VOTER I0517 06:46:08.050829 18675 consensus_peers.cc:456] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc -> Peer cb7678b41edd473396efaafe766253da (127.4.96.5:59858): Closing peer: cb7678b41edd473396efaafe766253da I0517 06:46:08.051283 18675 consensus_peers.cc:456] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc -> Peer 914aaa60c2844b42a89d7411a5c778b3 (127.4.96.4:39320): Closing peer: 914aaa60c2844b42a89d7411a5c778b3 I0517 06:46:08.051465 18675 consensus_peers.cc:456] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc -> Peer 976270186b154e39ac3c1d31e10e4988 (127.4.96.1:55449): Closing peer: 976270186b154e39ac3c1d31e10e4988 I0517 06:46:08.051838 18675 consensus_queue.cc:204] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 145, Committed index: 145, Last appended: 1.146, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: false } } I0517 06:46:08.056521 18026 raft_consensus.cc:1057] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 1 FOLLOWER]: Refusing update from remote peer bdb84e4dc496433f957e3188ed2d22dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 146. Preceding OpId from leader: term: 1 index: 147. (index mismatch) I0517 06:46:08.056743 18400 raft_consensus.cc:1057] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3 [term 1 LEARNER]: Refusing update from remote peer bdb84e4dc496433f957e3188ed2d22dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 146. Preceding OpId from leader: term: 1 index: 147. (index mismatch) I0517 06:46:08.057019 18524 raft_consensus.cc:1057] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 1 FOLLOWER]: Refusing update from remote peer bdb84e4dc496433f957e3188ed2d22dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 146. Preceding OpId from leader: term: 1 index: 147. (index mismatch) I0517 06:46:08.057725 18594 consensus_queue.cc:963] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 147, Last known committed idx: 145, Time since last communication: 0.000s I0517 06:46:08.058331 18620 consensus_queue.cc:963] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 147, Last known committed idx: 145, Time since last communication: 0.000s I0517 06:46:08.058681 18675 consensus_queue.cc:963] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 147, Last known committed idx: 145, Time since last communication: 0.000s I0517 06:46:08.064730 18620 raft_consensus.cc:2658] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 1 LEADER]: Committing config change with OpId 1.147: config changed from index 52 to 147, 914aaa60c2844b42a89d7411a5c778b3 (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: false } } } I0517 06:46:08.079939 17904 catalog_manager.cc:3686] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc reported cstate change: config changed from index 52 to 147, 914aaa60c2844b42a89d7411a5c778b3 (127.4.96.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bdb84e4dc496433f957e3188ed2d22dc" committed_config { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0517 06:46:08.084514 18026 raft_consensus.cc:2658] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 1 FOLLOWER]: Committing config change with OpId 1.147: config changed from index 52 to 147, 914aaa60c2844b42a89d7411a5c778b3 (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: false } } } I0517 06:46:08.094712 18524 raft_consensus.cc:2658] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 1 FOLLOWER]: Committing config change with OpId 1.147: config changed from index 52 to 147, 914aaa60c2844b42a89d7411a5c778b3 (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: false } } } I0517 06:46:08.097256 18400 raft_consensus.cc:2658] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3 [term 1 FOLLOWER]: Committing config change with OpId 1.147: config changed from index 52 to 147, 914aaa60c2844b42a89d7411a5c778b3 (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 147 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "976270186b154e39ac3c1d31e10e4988" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 55449 } attrs { replace: true } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: false } } } I0517 06:46:08.103898 17901 catalog_manager.cc:3437] Sending ChangeConfig:REMOVE_PEER on tablet e9da75b6930d48cebd69f0413be12078 (attempt 1) I0517 06:46:08.106132 18151 consensus_peers.cc:456] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc -> Peer cb7678b41edd473396efaafe766253da (127.4.96.5:59858): Closing peer: cb7678b41edd473396efaafe766253da I0517 06:46:08.106357 18151 consensus_peers.cc:456] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc -> Peer 914aaa60c2844b42a89d7411a5c778b3 (127.4.96.4:39320): Closing peer: 914aaa60c2844b42a89d7411a5c778b3 I0517 06:46:08.106532 18151 consensus_peers.cc:456] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc -> Peer 976270186b154e39ac3c1d31e10e4988 (127.4.96.1:55449): Closing peer: 976270186b154e39ac3c1d31e10e4988 I0517 06:46:08.106886 18151 consensus_queue.cc:204] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 147, Committed index: 147, Last appended: 1.148, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: false } } I0517 06:46:08.113263 18523 raft_consensus.cc:1057] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 1 FOLLOWER]: Refusing update from remote peer bdb84e4dc496433f957e3188ed2d22dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 147. Preceding OpId from leader: term: 1 index: 149. (index mismatch) I0517 06:46:08.113473 18399 raft_consensus.cc:1057] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3 [term 1 FOLLOWER]: Refusing update from remote peer bdb84e4dc496433f957e3188ed2d22dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 148. Preceding OpId from leader: term: 1 index: 149. (index mismatch) I0517 06:46:08.115288 18620 consensus_queue.cc:963] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 147, Time since last communication: 0.000s I0517 06:46:08.116309 18620 consensus_queue.cc:963] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 147, Time since last communication: 0.000s I0517 06:46:08.120527 18594 raft_consensus.cc:2658] T e9da75b6930d48cebd69f0413be12078 P bdb84e4dc496433f957e3188ed2d22dc [term 1 LEADER]: Committing config change with OpId 1.149: config changed from index 147 to 149, VOTER 976270186b154e39ac3c1d31e10e4988 (127.4.96.1) evicted. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: false } } } I0517 06:46:08.124102 18523 raft_consensus.cc:2658] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da [term 1 FOLLOWER]: Committing config change with OpId 1.149: config changed from index 147 to 149, VOTER 976270186b154e39ac3c1d31e10e4988 (127.4.96.1) evicted. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: false } } } I0517 06:46:08.131675 17886 catalog_manager.cc:3240] ChangeConfig:REMOVE_PEER RPC for tablet e9da75b6930d48cebd69f0413be12078 with cas_config_opid_index 147: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0517 06:46:08.131716 18400 raft_consensus.cc:2658] T e9da75b6930d48cebd69f0413be12078 P 914aaa60c2844b42a89d7411a5c778b3 [term 1 FOLLOWER]: Committing config change with OpId 1.149: config changed from index 147 to 149, VOTER 976270186b154e39ac3c1d31e10e4988 (127.4.96.1) evicted. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: false } } } I0517 06:46:08.134411 17903 catalog_manager.cc:3686] T e9da75b6930d48cebd69f0413be12078 P cb7678b41edd473396efaafe766253da reported cstate change: config changed from index 147 to 149, VOTER 976270186b154e39ac3c1d31e10e4988 (127.4.96.1) evicted. New cstate: current_term: 1 leader_uuid: "bdb84e4dc496433f957e3188ed2d22dc" committed_config { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "bdb84e4dc496433f957e3188ed2d22dc" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 39828 } } peers { permanent_uuid: "cb7678b41edd473396efaafe766253da" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 59858 } } peers { permanent_uuid: "914aaa60c2844b42a89d7411a5c778b3" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 39320 } attrs { promote: false } } } I0517 06:46:08.144786 17903 catalog_manager.cc:3013] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet e9da75b6930d48cebd69f0413be12078 on 976270186b154e39ac3c1d31e10e4988 (127.4.96.1:55449) (TS 976270186b154e39ac3c1d31e10e4988 not found in new config with opid_index 149) I0517 06:46:08.146157 18006 tablet_service.cc:800] Processing DeleteTablet for tablet e9da75b6930d48cebd69f0413be12078 with delete_type TABLET_DATA_TOMBSTONED (TS 976270186b154e39ac3c1d31e10e4988 not found in new config with opid_index 149) from {username='slave'} at 127.0.0.1:36803 I0517 06:46:08.158810 18789 tablet_replica.cc:261] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: stopping tablet replica I0517 06:46:08.159128 18789 maintenance_manager.cc:237] P 976270186b154e39ac3c1d31e10e4988: Unregistered op CompactRowSetsOp(e9da75b6930d48cebd69f0413be12078) I0517 06:46:08.159337 18789 maintenance_manager.cc:237] P 976270186b154e39ac3c1d31e10e4988: Unregistered op MinorDeltaCompactionOp(e9da75b6930d48cebd69f0413be12078) I0517 06:46:08.159525 18789 maintenance_manager.cc:237] P 976270186b154e39ac3c1d31e10e4988: Unregistered op MajorDeltaCompactionOp(e9da75b6930d48cebd69f0413be12078) I0517 06:46:08.159690 18789 maintenance_manager.cc:237] P 976270186b154e39ac3c1d31e10e4988: Unregistered op UndoDeltaBlockGCOp(e9da75b6930d48cebd69f0413be12078) I0517 06:46:08.159889 18789 maintenance_manager.cc:237] P 976270186b154e39ac3c1d31e10e4988: Unregistered op FlushMRSOp(e9da75b6930d48cebd69f0413be12078) I0517 06:46:08.160048 18789 maintenance_manager.cc:237] P 976270186b154e39ac3c1d31e10e4988: Unregistered op FlushDeltaMemStoresOp(e9da75b6930d48cebd69f0413be12078) I0517 06:46:08.160195 18789 maintenance_manager.cc:237] P 976270186b154e39ac3c1d31e10e4988: Unregistered op LogGCOp(e9da75b6930d48cebd69f0413be12078) I0517 06:46:08.160375 18789 raft_consensus.cc:2016] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 1 FOLLOWER]: Raft consensus shutting down. I0517 06:46:08.160639 18789 pending_rounds.cc:63] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: Trying to abort 1 pending transactions. I0517 06:46:08.160781 18789 pending_rounds.cc:68] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: Aborting transaction as it isn't in flight: id { term: 1 index: 148 } timestamp: 6252706070845022208 op_type: WRITE_OP write_request { tablet_id: "e9da75b6930d48cebd69f0413be12078" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\237[T:\010r\302\224\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "77271b3750d7a855" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6252706070660993024 } request_id { client_id: "378ec6309adc4059bf554f7047d55222" seq_no: 144 first_incomplete_seq_no: 144 attempt_no: 0 } I0517 06:46:08.162226 18789 raft_consensus.cc:2043] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988 [term 1 FOLLOWER]: Raft consensus is shut down! I0517 06:46:08.164840 18789 ts_tablet_manager.cc:1369] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0517 06:46:08.185230 18789 ts_tablet_manager.cc:1382] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.148 I0517 06:46:08.185489 18789 log.cc:981] T e9da75b6930d48cebd69f0413be12078 P 976270186b154e39ac3c1d31e10e4988: Deleting WAL directory at /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/wals/e9da75b6930d48cebd69f0413be12078 I0517 06:46:08.187021 17888 catalog_manager.cc:2991] TS 976270186b154e39ac3c1d31e10e4988 (127.4.96.1:55449): tablet e9da75b6930d48cebd69f0413be12078 (table TestTable [id=32ce2928a539415e9f8388736c762ee1]) successfully deleted Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- f764d804a93e440ba2cc8161931cf324 | 127.4.96.62:46703 | HEALTHY Tablet Server Summary UUID | Address | Status ----------------------------------+------------------+--------- 914aaa60c2844b42a89d7411a5c778b3 | 127.4.96.4:39320 | HEALTHY 976270186b154e39ac3c1d31e10e4988 | 127.4.96.1:55449 | HEALTHY bdb84e4dc496433f957e3188ed2d22dc | 127.4.96.2:39828 | HEALTHY c72d9ae7bc4d4d3b8e61e4d35a37b843 | 127.4.96.3:59498 | HEALTHY cb7678b41edd473396efaafe766253da | 127.4.96.5:59858 | HEALTHY Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 5 Tables | 1 Tablets | 1 Replicas | 3 OK I0517 06:46:09.556557 4480 log_verifier.cc:121] Checking tablet e9da75b6930d48cebd69f0413be12078 I0517 06:46:09.894943 4480 log_verifier.cc:172] Verified matching terms for 301 ops in tablet e9da75b6930d48cebd69f0413be12078 I0517 06:46:09.899515 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 17938 W0517 06:46:09.928128 17888 connection.cc:512] client connection to 127.4.96.1:55449 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:55449): Connection reset by peer (error 104) W0517 06:46:09.928275 18085 connection.cc:512] client connection to 127.4.96.1:55449 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:55449): Connection reset by peer (error 104) W0517 06:46:09.928990 17886 connection.cc:512] server connection from 127.4.96.1:36348 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:36348): Connection reset by peer (error 104) W0517 06:46:09.929216 18458 connection.cc:512] client connection to 127.4.96.1:55449 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:55449): Connection reset by peer (error 104) I0517 06:46:09.930223 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 18073 W0517 06:46:09.930152 17875 connection.cc:512] client connection to 127.4.96.1:55449 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:55449): Connection reset by peer (error 104) W0517 06:46:09.962378 17886 connection.cc:512] client connection to 127.4.96.2:39828 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:39828): Connection reset by peer (error 104) W0517 06:46:09.962412 18456 connection.cc:512] client connection to 127.4.96.2:39828 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:39828): Connection reset by peer (error 104) W0517 06:46:09.963161 18332 connection.cc:512] client connection to 127.4.96.2:39828 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:39828): Connection reset by peer (error 104) W0517 06:46:09.963884 17885 connection.cc:512] server connection from 127.4.96.2:59716 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:59716): Connection reset by peer (error 104) W0517 06:46:09.964392 17875 connection.cc:512] client connection to 127.4.96.2:39828 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:39828): Connection reset by peer (error 104) W0517 06:46:09.966190 18330 connection.cc:512] server connection from 127.4.96.2:34207 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:34207): Connection reset by peer (error 104) W0517 06:46:09.966203 18454 connection.cc:512] server connection from 127.4.96.2:37849 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:37849): Connection reset by peer (error 104) I0517 06:46:09.966900 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 18197 W0517 06:46:09.987655 17875 connection.cc:512] client connection to 127.4.96.3:59498 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:59498): Connection reset by peer (error 104) W0517 06:46:09.987745 17884 connection.cc:512] server connection from 127.4.96.3:45875 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:45875): Connection reset by peer (error 104) I0517 06:46:09.988468 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 18322 W0517 06:46:10.022145 17875 connection.cc:512] client connection to 127.4.96.4:39320 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:39320): Connection reset by peer (error 104) W0517 06:46:10.022140 17884 connection.cc:512] server connection from 127.4.96.4:42149 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:42149): Connection reset by peer (error 104) W0517 06:46:10.022898 17874 connection.cc:512] client connection to 127.4.96.4:39320 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:39320): Connection reset by peer (error 104) I0517 06:46:10.023491 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 18446 W0517 06:46:10.052325 17884 connection.cc:512] server connection from 127.4.96.5:46035 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:46035): Connection reset by peer (error 104) W0517 06:46:10.052429 17875 connection.cc:512] client connection to 127.4.96.5:59858 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:59858): Connection reset by peer (error 104) W0517 06:46:10.052872 17888 connection.cc:512] client connection to 127.4.96.5:59858 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:59858): Connection reset by peer (error 104) I0517 06:46:10.053225 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 17876 W0517 06:46:10.086518 17875 connection.cc:512] client connection to 127.4.96.62:46703 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:46703): Connection reset by peer (error 104) [ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/3 (7561 ms) [ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 I0517 06:46:10.104118 4480 test_util.cc:200] Using random seed: 818923295 I0517 06:46:10.106966 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:46:10.107065 4480 ts_itest-base.cc:114] -------------- I0517 06:46:10.107151 4480 ts_itest-base.cc:115] 5 tablet servers I0517 06:46:10.107234 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:46:10.107322 4480 ts_itest-base.cc:117] -------------- I0517 06:46:10.109259 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:60962 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:10.225742 18837 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:10.226094 18837 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:10.235318 18837 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:10.235492 18837 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:46:10.235615 18837 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:10.235725 18837 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:46:10.235862 18837 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:46:10.250098 18837 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:60962 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:10.257915 18837 master_main.cc:80] Initializing master server... I0517 06:46:10.259464 18837 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 184319us I0517 06:46:10.260246 18837 fs_manager.cc:260] Metadata directory not provided I0517 06:46:10.260391 18837 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:10.260607 18837 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:46:10.260733 18837 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:10.268062 18837 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "94f78b307d1e44af8983d4b0f54af370" format_stamp: "Formatted at 2018-05-17 06:46:10 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:10.268738 18837 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "94f78b307d1e44af8983d4b0f54af370" format_stamp: "Formatted at 2018-05-17 06:46:10 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:10.278281 18837 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.008s I0517 06:46:10.279424 18837 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:10.279829 18837 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:10.286245 18837 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0517 06:46:10.286367 18837 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "94f78b307d1e44af8983d4b0f54af370" format_stamp: "Formatted at 2018-05-17 06:46:10 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:10.286577 18837 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:10.476195 18837 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:10.477962 18837 master_main.cc:83] Starting Master server... I0517 06:46:10.517429 18837 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:60962 I0517 06:46:10.518287 18837 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:46:10.518388 18837 webserver.cc:184] Document root disabled I0517 06:46:10.519595 18837 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:37876/ I0517 06:46:10.520311 18837 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:46:10.522910 18898 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:10.523682 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 18837 I0517 06:46:10.524942 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60962 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:10.539371 18898 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370: Bootstrap starting. I0517 06:46:10.541589 18898 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370: No blocks or log segments found. Creating new log. I0517 06:46:10.542764 18898 log.cc:520] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370: Log is configured to *not* fsync() on all Append() calls I0517 06:46:10.545159 18898 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370: No bootstrap required, opened a new log I0517 06:46:10.557270 18898 raft_consensus.cc:303] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f78b307d1e44af8983d4b0f54af370" member_type: VOTER } I0517 06:46:10.557580 18898 raft_consensus.cc:329] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:46:10.557709 18898 raft_consensus.cc:604] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f78b307d1e44af8983d4b0f54af370, State: Initialized, Role: FOLLOWER I0517 06:46:10.558154 18898 consensus_queue.cc:227] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f78b307d1e44af8983d4b0f54af370" member_type: VOTER } I0517 06:46:10.558498 18898 raft_consensus.cc:343] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:46:10.558648 18898 raft_consensus.cc:435] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:46:10.558820 18898 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370: Snoozing failure detection for 1.630s (starting election) I0517 06:46:10.558964 18898 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:46:10.561410 18898 raft_consensus.cc:457] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f78b307d1e44af8983d4b0f54af370" member_type: VOTER } I0517 06:46:10.561825 18898 leader_election.cc:270] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:46:10.563495 18901 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370: Snoozing failure detection for 1.640s (election complete) I0517 06:46:10.563778 18901 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:46:10.570442 18901 raft_consensus.cc:571] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [term 1 LEADER]: Becoming Leader. State: Replica: 94f78b307d1e44af8983d4b0f54af370, State: Running, Role: LEADER I0517 06:46:10.570863 18901 consensus_queue.cc:204] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f78b307d1e44af8983d4b0f54af370" member_type: VOTER } I0517 06:46:10.574169 18898 sys_catalog.cc:335] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "94f78b307d1e44af8983d4b0f54af370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f78b307d1e44af8983d4b0f54af370" member_type: VOTER } } I0517 06:46:10.574604 18898 sys_catalog.cc:338] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [sys.catalog]: This master's current role is: LEADER I0517 06:46:10.577564 18898 sys_catalog.cc:422] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:46:10.577284 18903 sys_catalog.cc:335] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94f78b307d1e44af8983d4b0f54af370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f78b307d1e44af8983d4b0f54af370" member_type: VOTER } } I0517 06:46:10.664608 18903 sys_catalog.cc:338] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [sys.catalog]: This master's current role is: LEADER I0517 06:46:10.577718 18905 catalog_manager.cc:1001] Loading table and tablet metadata into memory... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:10.671684 18899 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:10.672149 18899 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:46:10.678503 18837 master_main.cc:86] Master server successfully started. I0517 06:46:10.578366 18902 sys_catalog.cc:335] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94f78b307d1e44af8983d4b0f54af370. Latest consensus state: current_term: 1 leader_uuid: "94f78b307d1e44af8983d4b0f54af370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f78b307d1e44af8983d4b0f54af370" member_type: VOTER } } I0517 06:46:10.678751 18902 sys_catalog.cc:338] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370 [sys.catalog]: This master's current role is: LEADER I0517 06:46:10.682013 18905 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... W0517 06:46:10.685866 18899 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:10.686116 18899 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:10.686575 18899 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 W0517 06:46:10.729554 18854 tablet.cc:1678] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:46:10.730409 18899 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60962 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:10.743094 18899 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:10.745321 18899 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 184319us I0517 06:46:10.745784 18899 fs_manager.cc:260] Metadata directory not provided I0517 06:46:10.745936 18899 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:10.746155 18899 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:46:10.746165 18905 catalog_manager.cc:899] Generated new certificate authority record I0517 06:46:10.746276 18899 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:10.747964 18905 catalog_manager.cc:1021] Loading token signing keys... I0517 06:46:10.753870 18899 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "a7176d67dfc24837bd6dd9f13bd9e294" format_stamp: "Formatted at 2018-05-17 06:46:10 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:10.754565 18899 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "a7176d67dfc24837bd6dd9f13bd9e294" format_stamp: "Formatted at 2018-05-17 06:46:10 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:10.757194 18905 catalog_manager.cc:3962] T 00000000000000000000000000000000 P 94f78b307d1e44af8983d4b0f54af370: Generated new TSK 0 I0517 06:46:10.764533 18899 fs_manager.cc:495] Time spent creating directory manager: real 0.010s user 0.004s sys 0.004s I0517 06:46:10.765688 18899 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:10.766122 18899 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:10.772660 18899 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:46:10.772775 18899 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "a7176d67dfc24837bd6dd9f13bd9e294" format_stamp: "Formatted at 2018-05-17 06:46:10 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:10.773000 18899 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:10.995033 18899 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:10.997591 18899 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:10.998780 18899 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:11.101289 18899 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:39007 I0517 06:46:11.102198 18899 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:46:11.102311 18899 webserver.cc:184] Document root disabled I0517 06:46:11.103678 18899 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:58327/ I0517 06:46:11.104374 18899 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:46:11.105515 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 18899 I0517 06:46:11.106855 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60962 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:11.112946 18899 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:11.130861 19030 heartbeater.cc:346] Connected to a master server at 127.4.96.62:60962 I0517 06:46:11.131105 19030 heartbeater.cc:426] Registering TS with master... I0517 06:46:11.131532 19030 heartbeater.cc:475] Master 127.4.96.62:60962 requested a full tablet report, sending... I0517 06:46:11.133072 18865 ts_manager.cc:81] Registered new tserver with Master: a7176d67dfc24837bd6dd9f13bd9e294 (127.4.96.1:39007) I0517 06:46:11.134770 18865 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:45958 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:11.267537 19034 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:11.268021 19034 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:11.282254 19034 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:11.282497 19034 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:11.282984 19034 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:46:11.303529 19034 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60962 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:11.316766 19034 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:11.319161 19034 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 184819us I0517 06:46:11.319762 19034 fs_manager.cc:260] Metadata directory not provided I0517 06:46:11.319926 19034 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:11.320183 19034 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:46:11.320341 19034 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:11.333199 19034 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "b4fad2b2ffa64aff93a8b1fd680412bf" format_stamp: "Formatted at 2018-05-17 06:46:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:11.334112 19034 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "b4fad2b2ffa64aff93a8b1fd680412bf" format_stamp: "Formatted at 2018-05-17 06:46:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:11.348279 19034 fs_manager.cc:495] Time spent creating directory manager: real 0.014s user 0.008s sys 0.008s I0517 06:46:11.349901 19034 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:11.350455 19034 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:11.360961 19034 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:46:11.361112 19034 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "b4fad2b2ffa64aff93a8b1fd680412bf" format_stamp: "Formatted at 2018-05-17 06:46:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:11.361385 19034 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:11.558748 19034 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:11.561858 19034 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:11.563012 19034 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:11.667368 19034 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:41019 I0517 06:46:11.668275 19034 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:46:11.668380 19034 webserver.cc:184] Document root disabled I0517 06:46:11.669744 19034 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:59285/ I0517 06:46:11.670450 19034 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:46:11.677254 19034 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:11.677588 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 19034 I0517 06:46:11.678620 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60962 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:11.700031 19155 heartbeater.cc:346] Connected to a master server at 127.4.96.62:60962 I0517 06:46:11.700371 19155 heartbeater.cc:426] Registering TS with master... I0517 06:46:11.700922 19155 heartbeater.cc:475] Master 127.4.96.62:60962 requested a full tablet report, sending... I0517 06:46:11.702611 18865 ts_manager.cc:81] Registered new tserver with Master: b4fad2b2ffa64aff93a8b1fd680412bf (127.4.96.2:41019) I0517 06:46:11.704232 18865 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:35032 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:11.788416 19159 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:11.788777 19159 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:11.796839 19159 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:11.797011 19159 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:11.797313 19159 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:46:11.810449 19159 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60962 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:11.818656 19159 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:11.820422 19159 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 184819us I0517 06:46:11.820860 19159 fs_manager.cc:260] Metadata directory not provided I0517 06:46:11.820982 19159 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:11.821178 19159 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:46:11.821290 19159 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:11.828861 19159 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "686e733774e04c40a9745d0f47f1cf15" format_stamp: "Formatted at 2018-05-17 06:46:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:11.829568 19159 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "686e733774e04c40a9745d0f47f1cf15" format_stamp: "Formatted at 2018-05-17 06:46:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:11.839148 19159 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:46:11.840281 19159 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:11.840684 19159 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:11.847249 19159 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0517 06:46:11.847363 19159 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "686e733774e04c40a9745d0f47f1cf15" format_stamp: "Formatted at 2018-05-17 06:46:11 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:11.847568 19159 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:12.048485 19159 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:12.051030 19159 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:12.052166 19159 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:12.137173 19030 heartbeater.cc:467] Master 127.4.96.62:60962 was elected leader, sending a full tablet report... I0517 06:46:12.137729 19159 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:46807 I0517 06:46:12.138921 19159 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:46:12.139060 19159 webserver.cc:184] Document root disabled I0517 06:46:12.140393 19159 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:47142/ I0517 06:46:12.141067 19159 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:46:12.143664 19159 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:12.145387 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 19159 I0517 06:46:12.146353 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60962 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:12.161957 19280 heartbeater.cc:346] Connected to a master server at 127.4.96.62:60962 I0517 06:46:12.162209 19280 heartbeater.cc:426] Registering TS with master... I0517 06:46:12.162663 19280 heartbeater.cc:475] Master 127.4.96.62:60962 requested a full tablet report, sending... I0517 06:46:12.163931 18865 ts_manager.cc:81] Registered new tserver with Master: 686e733774e04c40a9745d0f47f1cf15 (127.4.96.3:46807) I0517 06:46:12.165076 18865 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:40762 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:12.251644 19283 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:12.251960 19283 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:12.259789 19283 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:12.259945 19283 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:12.260258 19283 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:46:12.272439 19283 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60962 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:12.280349 19283 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:12.282089 19283 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 185319us I0517 06:46:12.282505 19283 fs_manager.cc:260] Metadata directory not provided I0517 06:46:12.282615 19283 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:12.282793 19283 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:46:12.282893 19283 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:12.290374 19283 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "a89407599158423fafb3e6890e501487" format_stamp: "Formatted at 2018-05-17 06:46:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:12.290998 19283 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "a89407599158423fafb3e6890e501487" format_stamp: "Formatted at 2018-05-17 06:46:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:12.300297 19283 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:46:12.301370 19283 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:12.301754 19283 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:12.308022 19283 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:46:12.308145 19283 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "a89407599158423fafb3e6890e501487" format_stamp: "Formatted at 2018-05-17 06:46:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:12.308333 19283 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:12.478688 19283 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:12.481231 19283 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:12.482508 19283 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:12.570437 19283 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:34231 I0517 06:46:12.571321 19283 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:46:12.571418 19283 webserver.cc:184] Document root disabled I0517 06:46:12.572599 19283 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:60873/ I0517 06:46:12.573277 19283 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:46:12.574100 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 19283 I0517 06:46:12.575341 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.4.96.5:0 --local_ip_for_outbound_sockets=127.4.96.5 --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60962 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:12.600049 19283 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:12.619765 19404 heartbeater.cc:346] Connected to a master server at 127.4.96.62:60962 I0517 06:46:12.620029 19404 heartbeater.cc:426] Registering TS with master... I0517 06:46:12.620465 19404 heartbeater.cc:475] Master 127.4.96.62:60962 requested a full tablet report, sending... I0517 06:46:12.621812 18865 ts_manager.cc:81] Registered new tserver with Master: a89407599158423fafb3e6890e501487 (127.4.96.4:34231) I0517 06:46:12.622928 18865 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:38130 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:12.696671 19407 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:12.696981 19407 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:12.704977 19407 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:12.705137 19407 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:12.705443 19407 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.5 I0517 06:46:12.707341 19155 heartbeater.cc:467] Master 127.4.96.62:60962 was elected leader, sending a full tablet report... I0517 06:46:12.717819 19407 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:60962 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.5 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:12.725790 19407 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:12.727536 19407 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 185319us I0517 06:46:12.727946 19407 fs_manager.cc:260] Metadata directory not provided I0517 06:46:12.728055 19407 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:12.728227 19407 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0517 06:46:12.728338 19407 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:12.735930 19407 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "dda0b84e770e42c7bbb9ab47a58906f4" format_stamp: "Formatted at 2018-05-17 06:46:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:12.736572 19407 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "dda0b84e770e42c7bbb9ab47a58906f4" format_stamp: "Formatted at 2018-05-17 06:46:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:12.745832 19407 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:46:12.746917 19407 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:12.747292 19407 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:12.753531 19407 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:46:12.753639 19407 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal uuid: "dda0b84e770e42c7bbb9ab47a58906f4" format_stamp: "Formatted at 2018-05-17 06:46:12 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:12.753851 19407 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:12.932199 19407 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:12.934522 19407 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:12.935454 19407 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:13.021450 19407 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.5:56986 I0517 06:46:13.022351 19407 webserver.cc:173] Starting webserver on 127.4.96.5:0 I0517 06:46:13.022462 19407 webserver.cc:184] Document root disabled I0517 06:46:13.023640 19407 webserver.cc:311] Webserver started. Bound to: http://127.4.96.5:46320/ I0517 06:46:13.024300 19407 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb I0517 06:46:13.029187 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 19407 I0517 06:46:13.033756 19407 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:13.046469 19528 heartbeater.cc:346] Connected to a master server at 127.4.96.62:60962 I0517 06:46:13.046772 19528 heartbeater.cc:426] Registering TS with master... I0517 06:46:13.047293 19528 heartbeater.cc:475] Master 127.4.96.62:60962 requested a full tablet report, sending... I0517 06:46:13.048868 18864 ts_manager.cc:81] Registered new tserver with Master: dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986) I0517 06:46:13.050662 18864 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.5:56357 I0517 06:46:13.052395 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:46:13.069046 18865 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55841: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:46:13.121034 19465 tablet_service.cc:759] Processing CreateTablet for tablet 550668a86bf14f46b733b9efc2f51f58 (table=TestTable [id=5cddf53db6264181bf0a6418406aa98b]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:46:13.121914 19465 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 550668a86bf14f46b733b9efc2f51f58. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:13.128065 18967 tablet_service.cc:759] Processing CreateTablet for tablet 550668a86bf14f46b733b9efc2f51f58 (table=TestTable [id=5cddf53db6264181bf0a6418406aa98b]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:46:13.128753 18967 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 550668a86bf14f46b733b9efc2f51f58. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:13.134434 19465 ts_tablet_manager.cc:1173] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4: Registered tablet (data state: TABLET_DATA_READY) I0517 06:46:13.137485 19217 tablet_service.cc:759] Processing CreateTablet for tablet 550668a86bf14f46b733b9efc2f51f58 (table=TestTable [id=5cddf53db6264181bf0a6418406aa98b]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:46:13.138300 19217 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 550668a86bf14f46b733b9efc2f51f58. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:13.138767 19548 ts_tablet_manager.cc:1028] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4: Bootstrapping tablet I0517 06:46:13.139255 19548 tablet_bootstrap.cc:436] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4: Bootstrap starting. I0517 06:46:13.139920 18967 ts_tablet_manager.cc:1173] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: Registered tablet (data state: TABLET_DATA_READY) I0517 06:46:13.141734 19549 ts_tablet_manager.cc:1028] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: Bootstrapping tablet I0517 06:46:13.142254 19549 tablet_bootstrap.cc:436] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: Bootstrap starting. I0517 06:46:13.142379 19548 tablet_bootstrap.cc:587] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4: No blocks or log segments found. Creating new log. I0517 06:46:13.143757 19548 log.cc:520] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4: Log is configured to *not* fsync() on all Append() calls I0517 06:46:13.146062 19549 tablet_bootstrap.cc:587] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: No blocks or log segments found. Creating new log. I0517 06:46:13.147671 19549 log.cc:520] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: Log is configured to *not* fsync() on all Append() calls I0517 06:46:13.151616 19548 tablet_bootstrap.cc:436] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4: No bootstrap required, opened a new log I0517 06:46:13.151904 19548 ts_tablet_manager.cc:1045] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s I0517 06:46:13.152180 19549 tablet_bootstrap.cc:436] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: No bootstrap required, opened a new log I0517 06:46:13.152570 19549 ts_tablet_manager.cc:1045] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s I0517 06:46:13.157433 19217 ts_tablet_manager.cc:1173] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15: Registered tablet (data state: TABLET_DATA_READY) I0517 06:46:13.159153 19552 ts_tablet_manager.cc:1028] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15: Bootstrapping tablet I0517 06:46:13.159680 19552 tablet_bootstrap.cc:436] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15: Bootstrap starting. I0517 06:46:13.163008 19552 tablet_bootstrap.cc:587] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15: No blocks or log segments found. Creating new log. I0517 06:46:13.164583 19552 log.cc:520] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15: Log is configured to *not* fsync() on all Append() calls I0517 06:46:13.164692 19548 raft_consensus.cc:303] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } } I0517 06:46:13.165510 19548 raft_consensus.cc:329] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:46:13.165711 19548 raft_consensus.cc:604] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda0b84e770e42c7bbb9ab47a58906f4, State: Initialized, Role: FOLLOWER I0517 06:46:13.166113 19549 raft_consensus.cc:303] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } } I0517 06:46:13.166759 19549 raft_consensus.cc:329] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:46:13.166348 19548 consensus_queue.cc:227] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } } I0517 06:46:13.166957 19549 raft_consensus.cc:604] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7176d67dfc24837bd6dd9f13bd9e294, State: Initialized, Role: FOLLOWER I0517 06:46:13.167446 19549 consensus_queue.cc:227] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } } I0517 06:46:13.178980 19528 heartbeater.cc:467] Master 127.4.96.62:60962 was elected leader, sending a full tablet report... I0517 06:46:13.184078 19549 ts_tablet_manager.cc:1073] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: Time spent starting tablet: real 0.031s user 0.016s sys 0.004s I0517 06:46:13.190608 19548 ts_tablet_manager.cc:1073] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4: Time spent starting tablet: real 0.039s user 0.008s sys 0.008s I0517 06:46:13.167696 19280 heartbeater.cc:467] Master 127.4.96.62:60962 was elected leader, sending a full tablet report... I0517 06:46:13.261550 19552 tablet_bootstrap.cc:436] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15: No bootstrap required, opened a new log I0517 06:46:13.261883 19552 ts_tablet_manager.cc:1045] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15: Time spent bootstrapping tablet: real 0.102s user 0.008s sys 0.092s I0517 06:46:13.271666 19552 raft_consensus.cc:303] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } } I0517 06:46:13.272219 19552 raft_consensus.cc:329] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:46:13.272372 19552 raft_consensus.cc:604] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 686e733774e04c40a9745d0f47f1cf15, State: Initialized, Role: FOLLOWER I0517 06:46:13.272773 19552 consensus_queue.cc:227] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } } I0517 06:46:13.274801 19552 ts_tablet_manager.cc:1073] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s W0517 06:46:13.283439 19529 tablet.cc:1678] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:46:13.323213 19554 raft_consensus.cc:435] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:46:13.361112 19554 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4: Snoozing failure detection for 2.146s (starting election) I0517 06:46:13.361383 19554 raft_consensus.cc:457] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } } I0517 06:46:13.362370 19554 leader_election.cc:243] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 686e733774e04c40a9745d0f47f1cf15 (127.4.96.3:46807) I0517 06:46:13.362807 19554 leader_election.cc:243] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a7176d67dfc24837bd6dd9f13bd9e294 (127.4.96.1:39007) I0517 06:46:13.372761 19237 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "550668a86bf14f46b733b9efc2f51f58" candidate_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686e733774e04c40a9745d0f47f1cf15" is_pre_election: true I0517 06:46:13.373353 19237 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15: Snoozing failure detection for 1.993s (vote granted) I0517 06:46:13.373620 19237 raft_consensus.cc:2219] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dda0b84e770e42c7bbb9ab47a58906f4 in term 0. I0517 06:46:13.374665 19415 leader_election.cc:379] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 686e733774e04c40a9745d0f47f1cf15 (127.4.96.3:46807) I0517 06:46:13.374863 19415 leader_election.cc:270] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:46:13.375288 19554 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4: Snoozing failure detection for 2.141s (election complete) I0517 06:46:13.375509 19554 raft_consensus.cc:2511] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:46:13.375701 19554 raft_consensus.cc:435] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:46:13.375880 19554 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4: Snoozing failure detection for 2.244s (starting election) I0517 06:46:13.376055 19554 raft_consensus.cc:2756] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:46:13.377488 18987 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "550668a86bf14f46b733b9efc2f51f58" candidate_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" is_pre_election: true I0517 06:46:13.378087 18987 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: Snoozing failure detection for 2.113s (vote granted) I0517 06:46:13.378351 18987 raft_consensus.cc:2219] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dda0b84e770e42c7bbb9ab47a58906f4 in term 0. I0517 06:46:13.379628 19416 leader_election.cc:379] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a7176d67dfc24837bd6dd9f13bd9e294 (127.4.96.1:39007) I0517 06:46:13.380230 19554 raft_consensus.cc:457] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } } I0517 06:46:13.380865 19554 leader_election.cc:243] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [CANDIDATE]: Term 1 election: Requesting vote from peer 686e733774e04c40a9745d0f47f1cf15 (127.4.96.3:46807) I0517 06:46:13.381150 19554 leader_election.cc:243] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [CANDIDATE]: Term 1 election: Requesting vote from peer a7176d67dfc24837bd6dd9f13bd9e294 (127.4.96.1:39007) I0517 06:46:13.382093 19237 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "550668a86bf14f46b733b9efc2f51f58" candidate_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686e733774e04c40a9745d0f47f1cf15" I0517 06:46:13.382258 18987 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "550668a86bf14f46b733b9efc2f51f58" candidate_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" I0517 06:46:13.382510 19237 raft_consensus.cc:2756] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:46:13.382568 18987 raft_consensus.cc:2756] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:46:13.382769 18987 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: Snoozing failure detection for 1.605s (vote granted) I0517 06:46:13.382750 19237 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15: Snoozing failure detection for 2.062s (vote granted) I0517 06:46:13.385438 18987 raft_consensus.cc:2219] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dda0b84e770e42c7bbb9ab47a58906f4 in term 1. I0517 06:46:13.386132 19416 leader_election.cc:379] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [CANDIDATE]: Term 1 election: Vote granted by peer a7176d67dfc24837bd6dd9f13bd9e294 (127.4.96.1:39007) I0517 06:46:13.386299 19416 leader_election.cc:270] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:46:13.386620 19554 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4: Snoozing failure detection for 1.548s (election complete) I0517 06:46:13.386804 19554 raft_consensus.cc:2511] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:46:13.386859 19237 raft_consensus.cc:2219] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dda0b84e770e42c7bbb9ab47a58906f4 in term 1. I0517 06:46:13.387768 19415 leader_election.cc:379] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [CANDIDATE]: Term 1 election: Vote granted by peer 686e733774e04c40a9745d0f47f1cf15 (127.4.96.3:46807) I0517 06:46:13.388185 19554 raft_consensus.cc:571] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [term 1 LEADER]: Becoming Leader. State: Replica: dda0b84e770e42c7bbb9ab47a58906f4, State: Running, Role: LEADER I0517 06:46:13.388607 19554 consensus_queue.cc:204] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } } I0517 06:46:13.393658 18864 catalog_manager.cc:3686] T 550668a86bf14f46b733b9efc2f51f58 P dda0b84e770e42c7bbb9ab47a58906f4 reported cstate change: term changed from 0 to 1, leader changed from to dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5). New cstate: current_term: 1 leader_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } health_report { overall_health: UNKNOWN } } } W0517 06:46:13.394773 19281 tablet.cc:1678] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:46:13.363960 19031 tablet.cc:1678] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:46:13.439855 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:46:13.441845 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a7176d67dfc24837bd6dd9f13bd9e294 to finish bootstrapping I0517 06:46:13.451459 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b4fad2b2ffa64aff93a8b1fd680412bf to finish bootstrapping I0517 06:46:13.461324 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 686e733774e04c40a9745d0f47f1cf15 to finish bootstrapping I0517 06:46:13.470331 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a89407599158423fafb3e6890e501487 to finish bootstrapping I0517 06:46:13.482990 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver dda0b84e770e42c7bbb9ab47a58906f4 to finish bootstrapping I0517 06:46:13.495437 4480 test_util.cc:200] Using random seed: 822314618 I0517 06:46:13.513149 4480 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0517 06:46:13.513720 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 19407 W0517 06:46:13.531812 18845 connection.cc:512] client connection to 127.4.96.5:56986 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:56986): Connection reset by peer (error 104) W0517 06:46:13.532999 18849 connection.cc:512] server connection from 127.4.96.5:56357 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:56357): Connection reset by peer (error 104) W0517 06:46:13.533028 18836 connection.cc:512] client connection to 127.4.96.5:56986 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:56986): Connection reset by peer (error 104) W0517 06:46:13.534165 19169 connection.cc:512] server connection from 127.4.96.5:36227 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:36227): Connection reset by peer (error 104) W0517 06:46:13.535198 18917 connection.cc:512] server connection from 127.4.96.5:36113 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:36113): Connection reset by peer (error 104) W0517 06:46:13.548038 19562 meta_cache.cc:216] Tablet 550668a86bf14f46b733b9efc2f51f58: Replica dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986) has failed: Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111) I0517 06:46:13.625262 19404 heartbeater.cc:467] Master 127.4.96.62:60962 was elected leader, sending a full tablet report... W0517 06:46:13.662469 19569 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:14.583941 19562 meta_cache.cc:216] Tablet 550668a86bf14f46b733b9efc2f51f58: Replica dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986) has failed: Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111) I0517 06:46:14.992139 19612 raft_consensus.cc:435] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:46:14.992339 19612 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: Snoozing failure detection for 1.777s (starting election) I0517 06:46:14.992523 19612 raft_consensus.cc:457] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } } I0517 06:46:14.993576 19612 leader_election.cc:243] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986) I0517 06:46:14.994648 19612 leader_election.cc:243] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 686e733774e04c40a9745d0f47f1cf15 (127.4.96.3:46807) W0517 06:46:15.007263 18917 leader_election.cc:299] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986): Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111) I0517 06:46:15.051501 19237 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "550668a86bf14f46b733b9efc2f51f58" candidate_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686e733774e04c40a9745d0f47f1cf15" is_pre_election: true I0517 06:46:15.051945 19237 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15: Snoozing failure detection for 1.605s (vote granted) I0517 06:46:15.052134 19237 raft_consensus.cc:2219] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7176d67dfc24837bd6dd9f13bd9e294 in term 1. I0517 06:46:15.060328 18917 leader_election.cc:379] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 686e733774e04c40a9745d0f47f1cf15 (127.4.96.3:46807) I0517 06:46:15.060621 18917 leader_election.cc:270] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0517 06:46:15.061219 19612 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: Snoozing failure detection for 1.801s (election complete) I0517 06:46:15.061432 19612 raft_consensus.cc:2511] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0517 06:46:15.061601 19612 raft_consensus.cc:435] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:46:15.061761 19612 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: Snoozing failure detection for 1.954s (starting election) I0517 06:46:15.061914 19612 raft_consensus.cc:2756] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:46:15.066053 19612 raft_consensus.cc:457] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } } I0517 06:46:15.066977 19612 leader_election.cc:243] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [CANDIDATE]: Term 2 election: Requesting vote from peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986) I0517 06:46:15.067358 19612 leader_election.cc:243] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [CANDIDATE]: Term 2 election: Requesting vote from peer 686e733774e04c40a9745d0f47f1cf15 (127.4.96.3:46807) W0517 06:46:15.069139 18917 leader_election.cc:299] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986): Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111) I0517 06:46:15.070026 19237 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "550668a86bf14f46b733b9efc2f51f58" candidate_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686e733774e04c40a9745d0f47f1cf15" I0517 06:46:15.070369 19237 raft_consensus.cc:2756] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15 [term 1 FOLLOWER]: Advancing to term 2 I0517 06:46:15.070544 19237 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15: Snoozing failure detection for 1.951s (vote granted) I0517 06:46:15.074566 19237 raft_consensus.cc:2219] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7176d67dfc24837bd6dd9f13bd9e294 in term 2. I0517 06:46:15.075917 18917 leader_election.cc:379] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [CANDIDATE]: Term 2 election: Vote granted by peer 686e733774e04c40a9745d0f47f1cf15 (127.4.96.3:46807) I0517 06:46:15.076167 18917 leader_election.cc:270] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0517 06:46:15.076639 19612 raft_consensus.cc:2704] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294: Snoozing failure detection for 2.030s (election complete) I0517 06:46:15.076862 19612 raft_consensus.cc:2511] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 2 FOLLOWER]: Leader election won for term 2 I0517 06:46:15.079723 19612 raft_consensus.cc:571] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [term 2 LEADER]: Becoming Leader. State: Replica: a7176d67dfc24837bd6dd9f13bd9e294, State: Running, Role: LEADER I0517 06:46:15.080209 19612 consensus_queue.cc:204] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } } I0517 06:46:15.086093 18865 catalog_manager.cc:3686] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 reported cstate change: term changed from 1 to 2, leader changed from dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5) to a7176d67dfc24837bd6dd9f13bd9e294 (127.4.96.1). New cstate: current_term: 2 leader_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda0b84e770e42c7bbb9ab47a58906f4" member_type: VOTER last_known_addr { host: "127.4.96.5" port: 56986 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7176d67dfc24837bd6dd9f13bd9e294" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 39007 } health_report { overall_health: HEALTHY } } } I0517 06:46:15.154633 19237 raft_consensus.cc:1057] T 550668a86bf14f46b733b9efc2f51f58 P 686e733774e04c40a9745d0f47f1cf15 [term 2 FOLLOWER]: Refusing update from remote peer a7176d67dfc24837bd6dd9f13bd9e294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch) W0517 06:46:15.154672 18917 consensus_peers.cc:434] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 -> Peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986): Couldn't send request to peer dda0b84e770e42c7bbb9ab47a58906f4 for tablet 550668a86bf14f46b733b9efc2f51f58. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0517 06:46:15.155776 19626 consensus_queue.cc:963] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "686e733774e04c40a9745d0f47f1cf15" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 46807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0517 06:46:15.611444 18917 consensus_peers.cc:434] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 -> Peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986): Couldn't send request to peer dda0b84e770e42c7bbb9ab47a58906f4 for tablet 550668a86bf14f46b733b9efc2f51f58. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0517 06:46:16.148435 18917 consensus_peers.cc:434] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 -> Peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986): Couldn't send request to peer dda0b84e770e42c7bbb9ab47a58906f4 for tablet 550668a86bf14f46b733b9efc2f51f58. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0517 06:46:16.566115 18917 consensus_peers.cc:434] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 -> Peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986): Couldn't send request to peer dda0b84e770e42c7bbb9ab47a58906f4 for tablet 550668a86bf14f46b733b9efc2f51f58. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0517 06:46:17.130208 18917 consensus_peers.cc:434] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 -> Peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986): Couldn't send request to peer dda0b84e770e42c7bbb9ab47a58906f4 for tablet 550668a86bf14f46b733b9efc2f51f58. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0517 06:46:17.492228 19172 debug-util.cc:397] Leaking SignalData structure 0x7b080002b080 after lost signal to thread 19160 W0517 06:46:17.683765 18917 consensus_peers.cc:434] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 -> Peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986): Couldn't send request to peer dda0b84e770e42c7bbb9ab47a58906f4 for tablet 550668a86bf14f46b733b9efc2f51f58. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0517 06:46:18.282763 18917 consensus_peers.cc:434] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 -> Peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986): Couldn't send request to peer dda0b84e770e42c7bbb9ab47a58906f4 for tablet 550668a86bf14f46b733b9efc2f51f58. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W0517 06:46:18.798353 18917 consensus_peers.cc:434] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 -> Peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986): Couldn't send request to peer dda0b84e770e42c7bbb9ab47a58906f4 for tablet 550668a86bf14f46b733b9efc2f51f58. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W0517 06:46:19.268788 18917 consensus_peers.cc:434] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 -> Peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986): Couldn't send request to peer dda0b84e770e42c7bbb9ab47a58906f4 for tablet 550668a86bf14f46b733b9efc2f51f58. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. Corruption: ksck pre-move health check failed: timed out with ksck errors remaining: last error: 1 out of 1 table(s) are not healthy W0517 06:46:19.890285 18917 consensus_peers.cc:434] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 -> Peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986): Couldn't send request to peer dda0b84e770e42c7bbb9ab47a58906f4 for tablet 550668a86bf14f46b733b9efc2f51f58. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. W0517 06:46:20.324499 18917 consensus_peers.cc:434] T 550668a86bf14f46b733b9efc2f51f58 P a7176d67dfc24837bd6dd9f13bd9e294 -> Peer dda0b84e770e42c7bbb9ab47a58906f4 (127.4.96.5:56986): Couldn't send request to peer dda0b84e770e42c7bbb9ab47a58906f4 for tablet 550668a86bf14f46b733b9efc2f51f58. Status: Network error: Client connection negotiation failed: client connection to 127.4.96.5:56986: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 11 times. I0517 06:46:20.582330 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 18899 I0517 06:46:20.618809 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 19034 W0517 06:46:20.618747 18846 connection.cc:512] client connection to 127.4.96.1:39007 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:39007): Connection reset by peer (error 104) W0517 06:46:20.620271 19167 connection.cc:512] server connection from 127.4.96.1:46214 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:46214): Connection reset by peer (error 104) W0517 06:46:20.630378 18836 connection.cc:512] client connection to 127.4.96.1:39007 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:39007): Connection reset by peer (error 104) W0517 06:46:20.631297 18845 connection.cc:512] server connection from 127.4.96.1:45958 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:45958): Connection reset by peer (error 104) W0517 06:46:20.644233 18836 connection.cc:512] client connection to 127.4.96.2:41019 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:41019): Connection reset by peer (error 104) W0517 06:46:20.644326 18849 connection.cc:512] server connection from 127.4.96.2:35032 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:35032): Connection reset by peer (error 104) I0517 06:46:20.645315 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 19159 W0517 06:46:20.680702 18845 connection.cc:512] client connection to 127.4.96.3:46807 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:46807): Connection reset by peer (error 104) I0517 06:46:20.680793 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 19283 W0517 06:46:20.682121 18845 connection.cc:512] server connection from 127.4.96.3:40762 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:40762): Connection reset by peer (error 104) W0517 06:46:20.682211 18836 connection.cc:512] client connection to 127.4.96.3:46807 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:46807): Connection reset by peer (error 104) W0517 06:46:20.706871 18836 connection.cc:512] client connection to 127.4.96.4:34231 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:34231): Connection reset by peer (error 104) I0517 06:46:20.707811 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 18837 W0517 06:46:20.734452 18836 connection.cc:512] client connection to 127.4.96.62:60962 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:60962): Connection reset by peer (error 104) [ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (10645 ms) [ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/5 I0517 06:46:20.749596 4480 test_util.cc:200] Using random seed: 829568775 I0517 06:46:20.752861 4480 ts_itest-base.cc:113] Starting cluster with: I0517 06:46:20.752961 4480 ts_itest-base.cc:114] -------------- I0517 06:46:20.753036 4480 ts_itest-base.cc:115] 5 tablet servers I0517 06:46:20.753111 4480 ts_itest-base.cc:116] 3 replicas per TS I0517 06:46:20.753182 4480 ts_itest-base.cc:117] -------------- I0517 06:46:20.755369 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:50667 --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:20.873517 19734 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:20.873847 19734 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:20.882869 19734 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:20.883026 19734 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0517 06:46:20.883133 19734 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:20.883225 19734 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0517 06:46:20.883335 19734 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0517 06:46:20.897188 19734 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.4.96.62:50667 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.4.96.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:20.905030 19734 master_main.cc:80] Initializing master server... I0517 06:46:20.906571 19734 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 189319us I0517 06:46:20.907333 19734 fs_manager.cc:260] Metadata directory not provided I0517 06:46:20.907456 19734 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:20.907646 19734 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0517 06:46:20.907753 19734 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:20.915066 19734 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/instance: uuid: "c40f380324a54488bb17ddd4ebdc8e69" format_stamp: "Formatted at 2018-05-17 06:46:20 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:20.915715 19734 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "c40f380324a54488bb17ddd4ebdc8e69" format_stamp: "Formatted at 2018-05-17 06:46:20 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:20.925278 19734 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0517 06:46:20.926425 19734 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:20.926837 19734 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:20.933239 19734 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:46:20.933346 19734 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/wal uuid: "c40f380324a54488bb17ddd4ebdc8e69" format_stamp: "Formatted at 2018-05-17 06:46:20 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:20.933538 19734 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:21.089709 19734 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:21.091472 19734 master_main.cc:83] Starting Master server... I0517 06:46:21.131685 19734 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.62:50667 I0517 06:46:21.132550 19734 webserver.cc:173] Starting webserver on 127.4.96.62:0 I0517 06:46:21.132658 19734 webserver.cc:184] Document root disabled I0517 06:46:21.133872 19734 webserver.cc:311] Webserver started. Bound to: http://127.4.96.62:47540/ I0517 06:46:21.134605 19734 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/master-0/data/info.pb I0517 06:46:21.137140 19795 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:21.141499 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master as pid 19734 I0517 06:46:21.142438 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.4.96.1:0 --local_ip_for_outbound_sockets=127.4.96.1 --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50667 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:21.150949 19795 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69: Bootstrap starting. I0517 06:46:21.153203 19795 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69: No blocks or log segments found. Creating new log. I0517 06:46:21.154428 19795 log.cc:520] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69: Log is configured to *not* fsync() on all Append() calls I0517 06:46:21.156818 19795 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69: No bootstrap required, opened a new log I0517 06:46:21.168495 19795 raft_consensus.cc:303] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40f380324a54488bb17ddd4ebdc8e69" member_type: VOTER } I0517 06:46:21.168807 19795 raft_consensus.cc:329] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:46:21.168965 19795 raft_consensus.cc:604] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40f380324a54488bb17ddd4ebdc8e69, State: Initialized, Role: FOLLOWER I0517 06:46:21.169450 19795 consensus_queue.cc:227] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40f380324a54488bb17ddd4ebdc8e69" member_type: VOTER } I0517 06:46:21.169813 19795 raft_consensus.cc:343] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0517 06:46:21.169988 19795 raft_consensus.cc:435] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0517 06:46:21.170157 19795 raft_consensus.cc:2704] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69: Snoozing failure detection for 2.127s (starting election) I0517 06:46:21.170321 19795 raft_consensus.cc:2756] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:46:21.172977 19795 raft_consensus.cc:457] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40f380324a54488bb17ddd4ebdc8e69" member_type: VOTER } I0517 06:46:21.173403 19795 leader_election.cc:270] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:46:21.175071 19798 raft_consensus.cc:2704] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69: Snoozing failure detection for 1.614s (election complete) I0517 06:46:21.175355 19798 raft_consensus.cc:2511] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:46:21.186406 19798 raft_consensus.cc:571] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [term 1 LEADER]: Becoming Leader. State: Replica: c40f380324a54488bb17ddd4ebdc8e69, State: Running, Role: LEADER I0517 06:46:21.186823 19798 consensus_queue.cc:204] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40f380324a54488bb17ddd4ebdc8e69" member_type: VOTER } I0517 06:46:21.190574 19795 sys_catalog.cc:335] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "c40f380324a54488bb17ddd4ebdc8e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40f380324a54488bb17ddd4ebdc8e69" member_type: VOTER } } I0517 06:46:21.191443 19795 sys_catalog.cc:338] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [sys.catalog]: This master's current role is: LEADER I0517 06:46:21.190660 19799 sys_catalog.cc:335] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c40f380324a54488bb17ddd4ebdc8e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40f380324a54488bb17ddd4ebdc8e69" member_type: VOTER } } I0517 06:46:21.251670 19799 sys_catalog.cc:338] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [sys.catalog]: This master's current role is: LEADER I0517 06:46:21.191216 19800 sys_catalog.cc:335] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c40f380324a54488bb17ddd4ebdc8e69. Latest consensus state: current_term: 1 leader_uuid: "c40f380324a54488bb17ddd4ebdc8e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40f380324a54488bb17ddd4ebdc8e69" member_type: VOTER } } I0517 06:46:21.253684 19800 sys_catalog.cc:338] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [sys.catalog]: This master's current role is: LEADER I0517 06:46:21.254940 19795 sys_catalog.cc:422] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69 [sys.catalog]: configured and running, proceeding with master startup. I0517 06:46:21.255239 19802 catalog_manager.cc:1001] Loading table and tablet metadata into memory... I0517 06:46:21.256564 19734 master_main.cc:86] Master server successfully started. I0517 06:46:21.259959 19802 catalog_manager.cc:1012] Initializing Kudu internal certificate authority... I0517 06:46:21.283196 19802 catalog_manager.cc:899] Generated new certificate authority record I0517 06:46:21.285493 19802 catalog_manager.cc:1021] Loading token signing keys... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:21.286886 19796 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:21.287406 19796 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:46:21.295770 19802 catalog_manager.cc:3962] T 00000000000000000000000000000000 P c40f380324a54488bb17ddd4ebdc8e69: Generated new TSK 0 W0517 06:46:21.299618 19796 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:21.299793 19796 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:21.300107 19796 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.1 I0517 06:46:21.316468 19796 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.4.96.1 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50667 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:21.324703 19796 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:21.326535 19796 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 189819us I0517 06:46:21.326989 19796 fs_manager.cc:260] Metadata directory not provided I0517 06:46:21.327114 19796 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:21.327324 19796 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0517 06:46:21.327438 19796 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:21.335012 19796 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "18153836475740a8b725615c36ea989a" format_stamp: "Formatted at 2018-05-17 06:46:21 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:21.335695 19796 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "18153836475740a8b725615c36ea989a" format_stamp: "Formatted at 2018-05-17 06:46:21 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:21.367149 19796 fs_manager.cc:495] Time spent creating directory manager: real 0.031s user 0.004s sys 0.028s I0517 06:46:21.368322 19796 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:21.368762 19796 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:21.379113 19796 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.000s sys 0.008s I0517 06:46:21.379253 19796 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal uuid: "18153836475740a8b725615c36ea989a" format_stamp: "Formatted at 2018-05-17 06:46:21 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:21.379474 19796 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:21.645016 19796 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:21.658977 19796 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:21.670262 19796 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:21.838825 19796 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.1:37343 I0517 06:46:21.839735 19796 webserver.cc:173] Starting webserver on 127.4.96.1:0 I0517 06:46:21.839839 19796 webserver.cc:184] Document root disabled I0517 06:46:21.841213 19796 webserver.cc:311] Webserver started. Bound to: http://127.4.96.1:34937/ I0517 06:46:21.841935 19796 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/data/info.pb I0517 06:46:21.842447 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 19796 I0517 06:46:21.843686 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.4.96.2:0 --local_ip_for_outbound_sockets=127.4.96.2 --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50667 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:21.851439 19796 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:21.870321 19927 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50667 I0517 06:46:21.870579 19927 heartbeater.cc:426] Registering TS with master... I0517 06:46:21.871028 19927 heartbeater.cc:475] Master 127.4.96.62:50667 requested a full tablet report, sending... I0517 06:46:21.872570 19762 ts_manager.cc:81] Registered new tserver with Master: 18153836475740a8b725615c36ea989a (127.4.96.1:37343) I0517 06:46:21.874233 19762 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.1:42891 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:22.013576 19931 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:22.014117 19931 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:22.028635 19931 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:22.028872 19931 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:22.029356 19931 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.2 I0517 06:46:22.050289 19931 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.4.96.2 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50667 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.2 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:22.063414 19931 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:22.065860 19931 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 190319us I0517 06:46:22.066489 19931 fs_manager.cc:260] Metadata directory not provided I0517 06:46:22.066659 19931 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:22.066926 19931 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0517 06:46:22.067091 19931 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:22.079865 19931 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "95acc391a60643c2a731f08a4617dfb1" format_stamp: "Formatted at 2018-05-17 06:46:22 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:22.080790 19931 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "95acc391a60643c2a731f08a4617dfb1" format_stamp: "Formatted at 2018-05-17 06:46:22 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:22.095691 19931 fs_manager.cc:495] Time spent creating directory manager: real 0.014s user 0.004s sys 0.012s I0517 06:46:22.097384 19931 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:22.097980 19931 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:22.108456 19931 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0517 06:46:22.108626 19931 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/wal uuid: "95acc391a60643c2a731f08a4617dfb1" format_stamp: "Formatted at 2018-05-17 06:46:22 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:22.108913 19931 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:22.330315 19931 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:22.332857 19931 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:22.334003 19931 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:22.447589 19931 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.2:48633 I0517 06:46:22.448503 19931 webserver.cc:173] Starting webserver on 127.4.96.2:0 I0517 06:46:22.448612 19931 webserver.cc:184] Document root disabled I0517 06:46:22.450031 19931 webserver.cc:311] Webserver started. Bound to: http://127.4.96.2:36901/ I0517 06:46:22.450724 19931 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-1/data/info.pb I0517 06:46:22.453557 19931 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:22.454586 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 19931 I0517 06:46:22.455826 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.4.96.3:0 --local_ip_for_outbound_sockets=127.4.96.3 --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50667 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:22.475329 20052 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50667 I0517 06:46:22.475575 20052 heartbeater.cc:426] Registering TS with master... I0517 06:46:22.476020 20052 heartbeater.cc:475] Master 127.4.96.62:50667 requested a full tablet report, sending... I0517 06:46:22.477406 19762 ts_manager.cc:81] Registered new tserver with Master: 95acc391a60643c2a731f08a4617dfb1 (127.4.96.2:48633) I0517 06:46:22.478533 19762 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.2:49050 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:22.616928 20056 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:22.617413 20056 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:22.631752 20056 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:22.631983 20056 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:22.632462 20056 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.3 I0517 06:46:22.652236 20056 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.4.96.3 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50667 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.3 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:22.664392 20056 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:22.666679 20056 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 190319us I0517 06:46:22.667249 20056 fs_manager.cc:260] Metadata directory not provided I0517 06:46:22.667408 20056 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:22.667646 20056 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0517 06:46:22.667784 20056 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:22.679440 20056 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "1bd6a295568e4e77af78531adbded176" format_stamp: "Formatted at 2018-05-17 06:46:22 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:22.680323 20056 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "1bd6a295568e4e77af78531adbded176" format_stamp: "Formatted at 2018-05-17 06:46:22 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:22.692159 20056 fs_manager.cc:495] Time spent creating directory manager: real 0.011s user 0.012s sys 0.000s I0517 06:46:22.693295 20056 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:22.693677 20056 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:22.699952 20056 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:46:22.700068 20056 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/wal uuid: "1bd6a295568e4e77af78531adbded176" format_stamp: "Formatted at 2018-05-17 06:46:22 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:22.700264 20056 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:22.876610 19927 heartbeater.cc:467] Master 127.4.96.62:50667 was elected leader, sending a full tablet report... I0517 06:46:22.897075 20056 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:22.899469 20056 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:22.900398 20056 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:22.985231 20056 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.3:53311 I0517 06:46:22.986135 20056 webserver.cc:173] Starting webserver on 127.4.96.3:0 I0517 06:46:22.986240 20056 webserver.cc:184] Document root disabled I0517 06:46:22.987421 20056 webserver.cc:311] Webserver started. Bound to: http://127.4.96.3:56008/ I0517 06:46:22.988121 20056 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-2/data/info.pb I0517 06:46:22.990115 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 20056 I0517 06:46:22.990862 20056 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:22.991190 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.4.96.4:0 --local_ip_for_outbound_sockets=127.4.96.4 --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50667 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:23.015844 20177 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50667 I0517 06:46:23.016086 20177 heartbeater.cc:426] Registering TS with master... I0517 06:46:23.016495 20177 heartbeater.cc:475] Master 127.4.96.62:50667 requested a full tablet report, sending... I0517 06:46:23.017944 19762 ts_manager.cc:81] Registered new tserver with Master: 1bd6a295568e4e77af78531adbded176 (127.4.96.3:53311) I0517 06:46:23.019582 19762 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.3:48296 WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:23.105224 20179 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:23.105540 20179 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:23.113492 20179 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:23.113647 20179 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:23.113950 20179 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.4 I0517 06:46:23.126204 20179 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.4.96.4 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50667 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.4 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:23.134408 20179 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:23.136183 20179 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 190819us I0517 06:46:23.136613 20179 fs_manager.cc:260] Metadata directory not provided I0517 06:46:23.136723 20179 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:23.136904 20179 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0517 06:46:23.137007 20179 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:23.144572 20179 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "4e9b544370e64ae897fd5c3979c23c0b" format_stamp: "Formatted at 2018-05-17 06:46:23 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:23.145226 20179 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "4e9b544370e64ae897fd5c3979c23c0b" format_stamp: "Formatted at 2018-05-17 06:46:23 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:23.154384 20179 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:46:23.155481 20179 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:23.155864 20179 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:23.162371 20179 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.004s I0517 06:46:23.162477 20179 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/wal uuid: "4e9b544370e64ae897fd5c3979c23c0b" format_stamp: "Formatted at 2018-05-17 06:46:23 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:23.162660 20179 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:23.322670 20179 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:23.325073 20179 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:23.326064 20179 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:23.413523 20179 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.4:56288 I0517 06:46:23.414436 20179 webserver.cc:173] Starting webserver on 127.4.96.4:0 I0517 06:46:23.414542 20179 webserver.cc:184] Document root disabled I0517 06:46:23.415767 20179 webserver.cc:311] Webserver started. Bound to: http://127.4.96.4:48051/ I0517 06:46:23.416450 20179 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-3/data/info.pb I0517 06:46:23.418237 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 20179 I0517 06:46:23.418932 20179 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:23.419226 4480 external_mini_cluster.cc:786] Running /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.4.96.5:0 --local_ip_for_outbound_sockets=127.4.96.5 --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50667 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0517 06:46:23.443249 20302 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50667 I0517 06:46:23.443480 20302 heartbeater.cc:426] Registering TS with master... I0517 06:46:23.443908 20302 heartbeater.cc:475] Master 127.4.96.62:50667 requested a full tablet report, sending... I0517 06:46:23.445262 19762 ts_manager.cc:81] Registered new tserver with Master: 4e9b544370e64ae897fd5c3979c23c0b (127.4.96.4:56288) I0517 06:46:23.446416 19762 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.4:35157 I0517 06:46:23.480756 20052 heartbeater.cc:467] Master 127.4.96.62:50667 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0517 06:46:23.525713 20304 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0517 06:46:23.526070 20304 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0517 06:46:23.533888 20304 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0517 06:46:23.534075 20304 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0517 06:46:23.534379 20304 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.96.5 I0517 06:46:23.546653 20304 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.4.96.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.4.96.5 --webserver_port=0 --tserver_master_addrs=127.4.96.62:50667 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.4.96.5 --log_dir=/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision f969af316286807a745fa39b9c2ee604c88b63c9 build type FASTDEBUG built by None at 17 May 2018 06:38:04 UTC on jenkins-slave-snjf build id 13499 TSAN enabled I0517 06:46:23.555001 20304 tablet_server_main.cc:85] Initializing tablet server... I0517 06:46:23.556761 20304 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 190819us I0517 06:46:23.557184 20304 fs_manager.cc:260] Metadata directory not provided I0517 06:46:23.557294 20304 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0517 06:46:23.557477 20304 server_base.cc:432] Could not load existing FS layout: Not found: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0517 06:46:23.557591 20304 server_base.cc:433] Attempting to create new FS layout instead I0517 06:46:23.565189 20304 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "543ffb18ca914ea2975ab4050d88728f" format_stamp: "Formatted at 2018-05-17 06:46:23 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:23.565836 20304 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "543ffb18ca914ea2975ab4050d88728f" format_stamp: "Formatted at 2018-05-17 06:46:23 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:23.575278 20304 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s I0517 06:46:23.576397 20304 env_posix.cc:1643] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0517 06:46:23.576794 20304 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0517 06:46:23.583168 20304 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0517 06:46:23.583273 20304 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/wal uuid: "543ffb18ca914ea2975ab4050d88728f" format_stamp: "Formatted at 2018-05-17 06:46:23 on dist-test-slave-dist-test-slave-26mv" I0517 06:46:23.583488 20304 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0517 06:46:23.760885 20304 env_posix.cc:1643] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0517 06:46:23.763212 20304 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0517 06:46:23.764117 20304 tablet_server_main.cc:90] Starting tablet server... I0517 06:46:23.848100 20304 rpc_server.cc:205] RPC server started. Bound to: 127.4.96.5:36843 I0517 06:46:23.848974 20304 webserver.cc:173] Starting webserver on 127.4.96.5:0 I0517 06:46:23.849073 20304 webserver.cc:184] Document root disabled I0517 06:46:23.850289 20304 webserver.cc:311] Webserver started. Bound to: http://127.4.96.5:34878/ I0517 06:46:23.850996 20304 server_base.cc:607] Dumped server information to /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-4/data/info.pb I0517 06:46:23.854167 20304 tablet_server_main.cc:93] Tablet server successfully started. I0517 06:46:23.856429 4480 external_mini_cluster.cc:848] Started /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver as pid 20304 I0517 06:46:23.872046 20426 heartbeater.cc:346] Connected to a master server at 127.4.96.62:50667 I0517 06:46:23.872293 20426 heartbeater.cc:426] Registering TS with master... I0517 06:46:23.872730 20426 heartbeater.cc:475] Master 127.4.96.62:50667 requested a full tablet report, sending... I0517 06:46:23.874233 19762 ts_manager.cc:81] Registered new tserver with Master: 543ffb18ca914ea2975ab4050d88728f (127.4.96.5:36843) I0517 06:46:23.875334 19762 master_service.cc:249] Signed X509 certificate for tserver {username='slave'} at 127.4.96.5:56320 I0517 06:46:23.877393 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:46:23.894289 19762 catalog_manager.cc:1315] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35584: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0517 06:46:23.933588 20114 tablet_service.cc:759] Processing CreateTablet for tablet 37de1e19ea3a42868d7b58490f06d7ec (table=TestTable [id=1c7a24332d034ba8b55fc1a67beaf90b]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:46:23.934526 20114 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 37de1e19ea3a42868d7b58490f06d7ec. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:23.940312 19989 tablet_service.cc:759] Processing CreateTablet for tablet 37de1e19ea3a42868d7b58490f06d7ec (table=TestTable [id=1c7a24332d034ba8b55fc1a67beaf90b]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:46:23.940312 19864 tablet_service.cc:759] Processing CreateTablet for tablet 37de1e19ea3a42868d7b58490f06d7ec (table=TestTable [id=1c7a24332d034ba8b55fc1a67beaf90b]), partition=RANGE (key) PARTITION UNBOUNDED I0517 06:46:23.941159 19864 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 37de1e19ea3a42868d7b58490f06d7ec. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:23.941177 19989 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 37de1e19ea3a42868d7b58490f06d7ec. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:23.953279 19989 ts_tablet_manager.cc:1173] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1: Registered tablet (data state: TABLET_DATA_READY) I0517 06:46:23.958740 19864 ts_tablet_manager.cc:1173] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: Registered tablet (data state: TABLET_DATA_READY) I0517 06:46:23.960855 20447 ts_tablet_manager.cc:1028] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: Bootstrapping tablet I0517 06:46:23.960911 20114 ts_tablet_manager.cc:1173] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: Registered tablet (data state: TABLET_DATA_READY) I0517 06:46:23.961300 20447 tablet_bootstrap.cc:436] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: Bootstrap starting. I0517 06:46:23.962599 20448 ts_tablet_manager.cc:1028] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: Bootstrapping tablet I0517 06:46:23.963079 20448 tablet_bootstrap.cc:436] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: Bootstrap starting. I0517 06:46:23.963730 20446 ts_tablet_manager.cc:1028] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1: Bootstrapping tablet I0517 06:46:23.964136 20447 tablet_bootstrap.cc:587] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: No blocks or log segments found. Creating new log. I0517 06:46:23.964592 20446 tablet_bootstrap.cc:436] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1: Bootstrap starting. I0517 06:46:23.965452 20447 log.cc:520] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: Log is configured to *not* fsync() on all Append() calls I0517 06:46:23.966074 20448 tablet_bootstrap.cc:587] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: No blocks or log segments found. Creating new log. I0517 06:46:23.967615 20448 log.cc:520] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: Log is configured to *not* fsync() on all Append() calls I0517 06:46:23.968122 20446 tablet_bootstrap.cc:587] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1: No blocks or log segments found. Creating new log. I0517 06:46:23.969650 20446 log.cc:520] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1: Log is configured to *not* fsync() on all Append() calls I0517 06:46:23.973690 20447 tablet_bootstrap.cc:436] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: No bootstrap required, opened a new log I0517 06:46:23.973786 20448 tablet_bootstrap.cc:436] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: No bootstrap required, opened a new log I0517 06:46:23.974016 20447 ts_tablet_manager.cc:1045] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s I0517 06:46:23.974066 20448 ts_tablet_manager.cc:1045] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s I0517 06:46:23.983994 20447 raft_consensus.cc:303] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } } I0517 06:46:23.984530 20447 raft_consensus.cc:329] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:46:23.984691 20447 raft_consensus.cc:604] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18153836475740a8b725615c36ea989a, State: Initialized, Role: FOLLOWER I0517 06:46:23.984916 20448 raft_consensus.cc:303] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } } I0517 06:46:23.985096 20447 consensus_queue.cc:227] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } } I0517 06:46:23.985488 20448 raft_consensus.cc:329] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:46:23.985651 20448 raft_consensus.cc:604] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd6a295568e4e77af78531adbded176, State: Initialized, Role: FOLLOWER I0517 06:46:23.986237 20446 tablet_bootstrap.cc:436] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1: No bootstrap required, opened a new log I0517 06:46:23.986517 20446 ts_tablet_manager.cc:1045] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.004s I0517 06:46:23.986076 20448 consensus_queue.cc:227] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } } I0517 06:46:23.988165 20177 heartbeater.cc:467] Master 127.4.96.62:50667 was elected leader, sending a full tablet report... I0517 06:46:23.994518 20448 ts_tablet_manager.cc:1073] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: Time spent starting tablet: real 0.020s user 0.016s sys 0.000s I0517 06:46:23.997381 20446 raft_consensus.cc:303] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } } I0517 06:46:23.997905 20446 raft_consensus.cc:329] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0517 06:46:23.998075 20446 raft_consensus.cc:604] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95acc391a60643c2a731f08a4617dfb1, State: Initialized, Role: FOLLOWER I0517 06:46:23.998486 20446 consensus_queue.cc:227] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } } I0517 06:46:24.000001 20447 ts_tablet_manager.cc:1073] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: Time spent starting tablet: real 0.026s user 0.008s sys 0.004s I0517 06:46:24.000890 20446 ts_tablet_manager.cc:1073] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s I0517 06:46:24.009282 20453 raft_consensus.cc:435] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0517 06:46:24.071318 20453 raft_consensus.cc:2704] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: Snoozing failure detection for 1.769s (starting election) I0517 06:46:24.071558 20453 raft_consensus.cc:457] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } } I0517 06:46:24.072535 20453 leader_election.cc:243] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 95acc391a60643c2a731f08a4617dfb1 (127.4.96.2:48633) I0517 06:46:24.072976 20453 leader_election.cc:243] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 18153836475740a8b725615c36ea989a (127.4.96.1:37343) I0517 06:46:24.083546 20009 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "37de1e19ea3a42868d7b58490f06d7ec" candidate_uuid: "1bd6a295568e4e77af78531adbded176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95acc391a60643c2a731f08a4617dfb1" is_pre_election: true I0517 06:46:24.084008 20009 raft_consensus.cc:2704] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1: Snoozing failure detection for 1.595s (vote granted) I0517 06:46:24.084205 20009 raft_consensus.cc:2219] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bd6a295568e4e77af78531adbded176 in term 0. I0517 06:46:24.085124 20068 leader_election.cc:379] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 95acc391a60643c2a731f08a4617dfb1 (127.4.96.2:48633) I0517 06:46:24.085356 20068 leader_election.cc:270] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0517 06:46:24.085409 19884 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "37de1e19ea3a42868d7b58490f06d7ec" candidate_uuid: "1bd6a295568e4e77af78531adbded176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18153836475740a8b725615c36ea989a" is_pre_election: true I0517 06:46:24.085906 20453 raft_consensus.cc:2704] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: Snoozing failure detection for 2.151s (election complete) I0517 06:46:24.085965 19884 raft_consensus.cc:2704] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: Snoozing failure detection for 1.553s (vote granted) I0517 06:46:24.086138 20453 raft_consensus.cc:2511] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0517 06:46:24.086203 19884 raft_consensus.cc:2219] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bd6a295568e4e77af78531adbded176 in term 0. I0517 06:46:24.086323 20453 raft_consensus.cc:435] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0517 06:46:24.086491 20453 raft_consensus.cc:2704] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: Snoozing failure detection for 2.001s (starting election) I0517 06:46:24.086671 20453 raft_consensus.cc:2756] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:46:24.087054 20068 leader_election.cc:379] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 18153836475740a8b725615c36ea989a (127.4.96.1:37343) I0517 06:46:24.089650 20453 raft_consensus.cc:457] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } } I0517 06:46:24.090306 20453 leader_election.cc:243] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [CANDIDATE]: Term 1 election: Requesting vote from peer 95acc391a60643c2a731f08a4617dfb1 (127.4.96.2:48633) I0517 06:46:24.090600 20453 leader_election.cc:243] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [CANDIDATE]: Term 1 election: Requesting vote from peer 18153836475740a8b725615c36ea989a (127.4.96.1:37343) I0517 06:46:24.091339 20009 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "37de1e19ea3a42868d7b58490f06d7ec" candidate_uuid: "1bd6a295568e4e77af78531adbded176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95acc391a60643c2a731f08a4617dfb1" I0517 06:46:24.091645 20009 raft_consensus.cc:2756] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [term 0 FOLLOWER]: Advancing to term 1 I0517 06:46:24.091851 20009 raft_consensus.cc:2704] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1: Snoozing failure detection for 1.506s (vote granted) I0517 06:46:24.091756 19884 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "37de1e19ea3a42868d7b58490f06d7ec" candidate_uuid: "1bd6a295568e4e77af78531adbded176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18153836475740a8b725615c36ea989a" I0517 06:46:24.092152 19884 raft_consensus.cc:2756] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [term 0 FOLLOWER]: Advancing to term 1 I0517 06:46:24.092389 19884 raft_consensus.cc:2704] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: Snoozing failure detection for 1.702s (vote granted) I0517 06:46:24.094909 20009 raft_consensus.cc:2219] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bd6a295568e4e77af78531adbded176 in term 1. I0517 06:46:24.096709 19884 raft_consensus.cc:2219] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bd6a295568e4e77af78531adbded176 in term 1. I0517 06:46:24.097223 20068 leader_election.cc:379] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [CANDIDATE]: Term 1 election: Vote granted by peer 95acc391a60643c2a731f08a4617dfb1 (127.4.96.2:48633) I0517 06:46:24.097437 20068 leader_election.cc:270] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0517 06:46:24.098455 20068 leader_election.cc:379] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [CANDIDATE]: Term 1 election: Vote granted by peer 18153836475740a8b725615c36ea989a (127.4.96.1:37343) I0517 06:46:24.098947 20453 raft_consensus.cc:2704] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: Snoozing failure detection for 1.815s (election complete) I0517 06:46:24.099175 20453 raft_consensus.cc:2511] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 1 FOLLOWER]: Leader election won for term 1 I0517 06:46:24.100383 20453 raft_consensus.cc:571] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 1 LEADER]: Becoming Leader. State: Replica: 1bd6a295568e4e77af78531adbded176, State: Running, Role: LEADER I0517 06:46:24.100901 20453 consensus_queue.cc:204] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } } I0517 06:46:24.109390 19760 catalog_manager.cc:3686] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 reported cstate change: term changed from 0 to 1, leader changed from to 1bd6a295568e4e77af78531adbded176 (127.4.96.3). New cstate: current_term: 1 leader_uuid: "1bd6a295568e4e77af78531adbded176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } health_report { overall_health: UNKNOWN } } } I0517 06:46:24.137353 4480 external_mini_cluster.cc:482] 5 TS(s) registered with all masters I0517 06:46:24.139442 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 18153836475740a8b725615c36ea989a to finish bootstrapping I0517 06:46:24.149346 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 95acc391a60643c2a731f08a4617dfb1 to finish bootstrapping I0517 06:46:24.159314 4480 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1bd6a295568e4e77af78531adbded176 to finish bootstrapping I0517 06:46:24.169121 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4e9b544370e64ae897fd5c3979c23c0b to finish bootstrapping I0517 06:46:24.181308 4480 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 543ffb18ca914ea2975ab4050d88728f to finish bootstrapping W0517 06:46:24.102355 19928 tablet.cc:1678] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: Can't schedule compaction. Clean time has not been advanced past its initial value. I0517 06:46:24.191520 4480 test_util.cc:200] Using random seed: 833010702 I0517 06:46:24.207546 4480 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0517 06:46:24.208127 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 20304 W0517 06:46:24.210160 20053 tablet.cc:1678] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:46:24.228615 19733 connection.cc:512] client connection to 127.4.96.5:36843 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:36843): Connection reset by peer (error 104) W0517 06:46:24.228641 19743 connection.cc:512] server connection from 127.4.96.5:56320 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.5:56320): Connection reset by peer (error 104) I0517 06:46:24.231717 19884 raft_consensus.cc:1057] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [term 1 FOLLOWER]: Refusing update from remote peer 1bd6a295568e4e77af78531adbded176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0517 06:46:24.231938 20009 raft_consensus.cc:1057] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [term 1 FOLLOWER]: Refusing update from remote peer 1bd6a295568e4e77af78531adbded176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0517 06:46:24.233134 20453 consensus_queue.cc:963] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0517 06:46:24.233677 20457 consensus_queue.cc:963] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0517 06:46:24.241434 20178 tablet.cc:1678] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: Can't schedule compaction. Clean time has not been advanced past its initial value. W0517 06:46:24.419760 20473 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0517 06:46:24.448752 20302 heartbeater.cc:467] Master 127.4.96.62:50667 was elected leader, sending a full tablet report... I0517 06:46:24.868186 20134 consensus_peers.cc:456] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 -> Peer 18153836475740a8b725615c36ea989a (127.4.96.1:37343): Closing peer: 18153836475740a8b725615c36ea989a I0517 06:46:24.868360 20134 consensus_peers.cc:456] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 -> Peer 95acc391a60643c2a731f08a4617dfb1 (127.4.96.2:48633): Closing peer: 95acc391a60643c2a731f08a4617dfb1 I0517 06:46:24.868613 20134 consensus_queue.cc:204] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.56, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: true } } I0517 06:46:24.874619 19884 raft_consensus.cc:1057] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [term 1 FOLLOWER]: Refusing update from remote peer 1bd6a295568e4e77af78531adbded176: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch) I0517 06:46:24.875000 20009 raft_consensus.cc:1057] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [term 1 FOLLOWER]: Refusing update from remote peer 1bd6a295568e4e77af78531adbded176: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch) I0517 06:46:24.876212 20477 consensus_queue.cc:963] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 55, Time since last communication: 0.000s I0517 06:46:24.876667 20457 consensus_queue.cc:963] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 55, Time since last communication: 0.000s W0517 06:46:24.881330 20065 consensus_peers.cc:434] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 -> Peer 4e9b544370e64ae897fd5c3979c23c0b (127.4.96.4:56288): Couldn't send request to peer 4e9b544370e64ae897fd5c3979c23c0b for tablet 37de1e19ea3a42868d7b58490f06d7ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37de1e19ea3a42868d7b58490f06d7ec. Retrying in the next heartbeat period. Already tried 1 times. I0517 06:46:24.880833 20457 raft_consensus.cc:2658] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 1 LEADER]: Committing config change with OpId 1.57: config changed from index -1 to 57, NON_VOTER 4e9b544370e64ae897fd5c3979c23c0b (127.4.96.4) added. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: true } } } I0517 06:46:24.883219 19884 raft_consensus.cc:2658] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [term 1 FOLLOWER]: Committing config change with OpId 1.57: config changed from index -1 to 57, NON_VOTER 4e9b544370e64ae897fd5c3979c23c0b (127.4.96.4) added. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: true } } } I0517 06:46:24.888922 19761 catalog_manager.cc:3686] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 reported cstate change: config changed from index -1 to 57, NON_VOTER 4e9b544370e64ae897fd5c3979c23c0b (127.4.96.4) added. New cstate: current_term: 1 leader_uuid: "1bd6a295568e4e77af78531adbded176" committed_config { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0517 06:46:24.889008 20009 raft_consensus.cc:2658] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [term 1 FOLLOWER]: Committing config change with OpId 1.57: config changed from index -1 to 57, NON_VOTER 4e9b544370e64ae897fd5c3979c23c0b (127.4.96.4) added. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: true } } } I0517 06:46:25.338923 20576 ts_tablet_manager.cc:669] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b: Initiating tablet copy from peer 1bd6a295568e4e77af78531adbded176 (127.4.96.3:53311) I0517 06:46:25.339275 20576 tablet_copy_client.cc:239] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.96.3:53311 I0517 06:46:25.351775 20154 tablet_copy_service.cc:133] P 1bd6a295568e4e77af78531adbded176: Received BeginTabletCopySession request for tablet 37de1e19ea3a42868d7b58490f06d7ec from peer 4e9b544370e64ae897fd5c3979c23c0b ({username='slave'} at 127.4.96.4:60894) I0517 06:46:25.352108 20154 tablet_copy_service.cc:154] P 1bd6a295568e4e77af78531adbded176: Beginning new tablet copy session on tablet 37de1e19ea3a42868d7b58490f06d7ec from peer 4e9b544370e64ae897fd5c3979c23c0b at {username='slave'} at 127.4.96.4:60894: session id = 4e9b544370e64ae897fd5c3979c23c0b-37de1e19ea3a42868d7b58490f06d7ec I0517 06:46:25.356050 20154 tablet_copy_source_session.cc:202] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: Tablet Copy: opened 0 blocks and 1 log segments I0517 06:46:25.359230 20576 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 37de1e19ea3a42868d7b58490f06d7ec. 1 dirs total, 0 dirs full, 0 dirs failed I0517 06:46:25.375476 20576 ts_tablet_manager.cc:1173] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b: Registered tablet (data state: TABLET_DATA_COPYING) I0517 06:46:25.376178 20576 tablet_copy_client.cc:565] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b: tablet copy: Starting download of 0 data blocks... I0517 06:46:25.376459 20576 tablet_copy_client.cc:528] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b: tablet copy: Starting download of 1 WAL segments... I0517 06:46:25.380219 20576 tablet_copy_client.cc:404] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0517 06:46:25.384528 20576 ts_tablet_manager.cc:1028] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b: Bootstrapping tablet I0517 06:46:25.384984 20576 tablet_bootstrap.cc:436] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b: Bootstrap starting. I0517 06:46:25.420589 20576 log.cc:520] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b: Log is configured to *not* fsync() on all Append() calls I0517 06:46:25.580425 20576 tablet_bootstrap.cc:436] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=99 overwritten=0 applied=98 ignored=0} inserts{seen=96 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0517 06:46:25.581797 20576 tablet_bootstrap.cc:436] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b: Bootstrap complete. I0517 06:46:25.582340 20576 ts_tablet_manager.cc:1045] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b: Time spent bootstrapping tablet: real 0.198s user 0.164s sys 0.012s I0517 06:46:25.593197 20576 raft_consensus.cc:303] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b [term 1 LEARNER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: true } } I0517 06:46:25.600430 20576 raft_consensus.cc:604] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e9b544370e64ae897fd5c3979c23c0b, State: Initialized, Role: LEARNER I0517 06:46:25.600909 20576 consensus_queue.cc:227] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 98, Last appended: 1.99, Last appended by leader: 99, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: NON_VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: true } } I0517 06:46:25.612203 20576 ts_tablet_manager.cc:1073] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b: Time spent starting tablet: real 0.030s user 0.016s sys 0.008s I0517 06:46:25.613428 20154 tablet_copy_service.cc:323] P 1bd6a295568e4e77af78531adbded176: Request end of tablet copy session 4e9b544370e64ae897fd5c3979c23c0b-37de1e19ea3a42868d7b58490f06d7ec received from {username='slave'} at 127.4.96.4:60894 I0517 06:46:25.613665 20154 tablet_copy_service.cc:411] P 1bd6a295568e4e77af78531adbded176: Ending tablet copy session 4e9b544370e64ae897fd5c3979c23c0b-37de1e19ea3a42868d7b58490f06d7ec on tablet 37de1e19ea3a42868d7b58490f06d7ec with peer 4e9b544370e64ae897fd5c3979c23c0b I0517 06:46:25.764273 20259 raft_consensus.cc:1002] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.56->[1.57-1.132] Dedup: 1.99->[1.100-1.132] I0517 06:46:25.880684 20453 raft_consensus.cc:866] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176: attempting to promote NON_VOTER 4e9b544370e64ae897fd5c3979c23c0b to VOTER I0517 06:46:25.881613 20453 consensus_peers.cc:456] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 -> Peer 4e9b544370e64ae897fd5c3979c23c0b (127.4.96.4:56288): Closing peer: 4e9b544370e64ae897fd5c3979c23c0b I0517 06:46:25.881799 20453 consensus_peers.cc:456] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 -> Peer 18153836475740a8b725615c36ea989a (127.4.96.1:37343): Closing peer: 18153836475740a8b725615c36ea989a I0517 06:46:25.882136 20453 consensus_peers.cc:456] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 -> Peer 95acc391a60643c2a731f08a4617dfb1 (127.4.96.2:48633): Closing peer: 95acc391a60643c2a731f08a4617dfb1 I0517 06:46:25.882524 20453 consensus_queue.cc:204] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 139, Committed index: 139, Last appended: 1.140, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: false } } I0517 06:46:25.888355 20258 raft_consensus.cc:1057] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b [term 1 LEARNER]: Refusing update from remote peer 1bd6a295568e4e77af78531adbded176: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 141. (index mismatch) I0517 06:46:25.888643 20009 raft_consensus.cc:1057] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [term 1 FOLLOWER]: Refusing update from remote peer 1bd6a295568e4e77af78531adbded176: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 141. (index mismatch) I0517 06:46:25.888741 19884 raft_consensus.cc:1057] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [term 1 FOLLOWER]: Refusing update from remote peer 1bd6a295568e4e77af78531adbded176: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 141. (index mismatch) I0517 06:46:25.889940 20453 consensus_queue.cc:963] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 139, Time since last communication: 0.000s I0517 06:46:25.891788 20457 consensus_queue.cc:963] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 138, Time since last communication: 0.000s I0517 06:46:25.892807 20457 consensus_queue.cc:963] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 139, Time since last communication: 0.000s I0517 06:46:25.906533 20453 raft_consensus.cc:2658] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 1 LEADER]: Committing config change with OpId 1.141: config changed from index 57 to 141, 4e9b544370e64ae897fd5c3979c23c0b (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: false } } } I0517 06:46:25.912305 19884 raft_consensus.cc:2658] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index 57 to 141, 4e9b544370e64ae897fd5c3979c23c0b (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: false } } } I0517 06:46:25.914726 20259 raft_consensus.cc:2658] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index 57 to 141, 4e9b544370e64ae897fd5c3979c23c0b (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: false } } } I0517 06:46:25.922689 19761 catalog_manager.cc:3686] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b reported cstate change: config changed from index 57 to 141, 4e9b544370e64ae897fd5c3979c23c0b (127.4.96.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1bd6a295568e4e77af78531adbded176" committed_config { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: false } } } I0517 06:46:25.925858 20009 raft_consensus.cc:2658] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index 57 to 141, 4e9b544370e64ae897fd5c3979c23c0b (127.4.96.4) changed from NON_VOTER to VOTER. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "18153836475740a8b725615c36ea989a" member_type: VOTER last_known_addr { host: "127.4.96.1" port: 37343 } attrs { replace: true } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: false } } } I0517 06:46:25.946321 19759 catalog_manager.cc:3437] Sending ChangeConfig:REMOVE_PEER on tablet 37de1e19ea3a42868d7b58490f06d7ec (attempt 1) I0517 06:46:25.950583 20134 consensus_peers.cc:456] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 -> Peer 4e9b544370e64ae897fd5c3979c23c0b (127.4.96.4:56288): Closing peer: 4e9b544370e64ae897fd5c3979c23c0b I0517 06:46:25.950752 20134 consensus_peers.cc:456] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 -> Peer 18153836475740a8b725615c36ea989a (127.4.96.1:37343): Closing peer: 18153836475740a8b725615c36ea989a I0517 06:46:25.950873 20134 consensus_peers.cc:456] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 -> Peer 95acc391a60643c2a731f08a4617dfb1 (127.4.96.2:48633): Closing peer: 95acc391a60643c2a731f08a4617dfb1 I0517 06:46:25.951344 20134 consensus_queue.cc:204] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 142, Committed index: 142, Last appended: 1.142, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: false } } I0517 06:46:25.954397 20009 raft_consensus.cc:1057] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [term 1 FOLLOWER]: Refusing update from remote peer 1bd6a295568e4e77af78531adbded176: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 143. (index mismatch) I0517 06:46:25.955157 20453 consensus_queue.cc:963] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s I0517 06:46:25.956285 20259 raft_consensus.cc:1057] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b [term 1 FOLLOWER]: Refusing update from remote peer 1bd6a295568e4e77af78531adbded176: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 143. (index mismatch) I0517 06:46:25.957175 20453 consensus_queue.cc:963] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s I0517 06:46:25.959172 20477 raft_consensus.cc:2658] T 37de1e19ea3a42868d7b58490f06d7ec P 1bd6a295568e4e77af78531adbded176 [term 1 LEADER]: Committing config change with OpId 1.143: config changed from index 141 to 143, VOTER 18153836475740a8b725615c36ea989a (127.4.96.1) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: false } } } I0517 06:46:25.960752 20009 raft_consensus.cc:2658] T 37de1e19ea3a42868d7b58490f06d7ec P 95acc391a60643c2a731f08a4617dfb1 [term 1 FOLLOWER]: Committing config change with OpId 1.143: config changed from index 141 to 143, VOTER 18153836475740a8b725615c36ea989a (127.4.96.1) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: false } } } I0517 06:46:25.962810 20259 raft_consensus.cc:2658] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b [term 1 FOLLOWER]: Committing config change with OpId 1.143: config changed from index 141 to 143, VOTER 18153836475740a8b725615c36ea989a (127.4.96.1) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: false } } } I0517 06:46:25.964778 19746 catalog_manager.cc:3240] ChangeConfig:REMOVE_PEER RPC for tablet 37de1e19ea3a42868d7b58490f06d7ec with cas_config_opid_index 141: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0517 06:46:25.968551 19762 catalog_manager.cc:3686] T 37de1e19ea3a42868d7b58490f06d7ec P 4e9b544370e64ae897fd5c3979c23c0b reported cstate change: config changed from index 141 to 143, VOTER 18153836475740a8b725615c36ea989a (127.4.96.1) evicted. New cstate: current_term: 1 leader_uuid: "1bd6a295568e4e77af78531adbded176" committed_config { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "1bd6a295568e4e77af78531adbded176" member_type: VOTER last_known_addr { host: "127.4.96.3" port: 53311 } } peers { permanent_uuid: "95acc391a60643c2a731f08a4617dfb1" member_type: VOTER last_known_addr { host: "127.4.96.2" port: 48633 } } peers { permanent_uuid: "4e9b544370e64ae897fd5c3979c23c0b" member_type: VOTER last_known_addr { host: "127.4.96.4" port: 56288 } attrs { promote: false } } } I0517 06:46:25.973335 19762 catalog_manager.cc:3013] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 37de1e19ea3a42868d7b58490f06d7ec on 18153836475740a8b725615c36ea989a (127.4.96.1:37343) (TS 18153836475740a8b725615c36ea989a not found in new config with opid_index 143) I0517 06:46:25.977900 19864 tablet_service.cc:800] Processing DeleteTablet for tablet 37de1e19ea3a42868d7b58490f06d7ec with delete_type TABLET_DATA_TOMBSTONED (TS 18153836475740a8b725615c36ea989a not found in new config with opid_index 143) from {username='slave'} at 127.0.0.1:59148 I0517 06:46:25.979578 20638 tablet_replica.cc:261] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: stopping tablet replica I0517 06:46:25.979794 20638 maintenance_manager.cc:237] P 18153836475740a8b725615c36ea989a: Unregistered op CompactRowSetsOp(37de1e19ea3a42868d7b58490f06d7ec) I0517 06:46:25.979939 20638 maintenance_manager.cc:237] P 18153836475740a8b725615c36ea989a: Unregistered op MinorDeltaCompactionOp(37de1e19ea3a42868d7b58490f06d7ec) I0517 06:46:25.980068 20638 maintenance_manager.cc:237] P 18153836475740a8b725615c36ea989a: Unregistered op MajorDeltaCompactionOp(37de1e19ea3a42868d7b58490f06d7ec) I0517 06:46:25.980190 20638 maintenance_manager.cc:237] P 18153836475740a8b725615c36ea989a: Unregistered op UndoDeltaBlockGCOp(37de1e19ea3a42868d7b58490f06d7ec) I0517 06:46:25.980360 20638 maintenance_manager.cc:237] P 18153836475740a8b725615c36ea989a: Unregistered op FlushMRSOp(37de1e19ea3a42868d7b58490f06d7ec) I0517 06:46:25.980541 20638 maintenance_manager.cc:237] P 18153836475740a8b725615c36ea989a: Unregistered op FlushDeltaMemStoresOp(37de1e19ea3a42868d7b58490f06d7ec) I0517 06:46:25.980710 20638 maintenance_manager.cc:237] P 18153836475740a8b725615c36ea989a: Unregistered op LogGCOp(37de1e19ea3a42868d7b58490f06d7ec) I0517 06:46:25.980870 20638 raft_consensus.cc:2016] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [term 1 FOLLOWER]: Raft consensus shutting down. I0517 06:46:25.981142 20638 raft_consensus.cc:2043] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a [term 1 FOLLOWER]: Raft consensus is shut down! I0517 06:46:25.983443 20638 ts_tablet_manager.cc:1369] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0517 06:46:26.000000 20638 ts_tablet_manager.cc:1382] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.142 I0517 06:46:26.000190 20638 log.cc:981] T 37de1e19ea3a42868d7b58490f06d7ec P 18153836475740a8b725615c36ea989a: Deleting WAL directory at /tmp/dist-test-task4QLboq/test-tmp/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1526539434354140-4480/raft_consensus-itest-cluster/ts-0/wal/wals/37de1e19ea3a42868d7b58490f06d7ec I0517 06:46:26.001591 19746 catalog_manager.cc:2991] TS 18153836475740a8b725615c36ea989a (127.4.96.1:37343): tablet 37de1e19ea3a42868d7b58490f06d7ec (table TestTable [id=1c7a24332d034ba8b55fc1a67beaf90b]) successfully deleted I0517 06:46:27.225272 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 19796 W0517 06:46:27.253193 19746 connection.cc:512] client connection to 127.4.96.1:37343 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:37343): Connection reset by peer (error 104) W0517 06:46:27.253235 20068 connection.cc:512] client connection to 127.4.96.1:37343 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:37343): Connection reset by peer (error 104) W0517 06:46:27.253865 19746 connection.cc:512] server connection from 127.4.96.1:42891 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:42891): Connection reset by peer (error 104) W0517 06:46:27.254355 19733 connection.cc:512] client connection to 127.4.96.1:37343 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.1:37343): Connection reset by peer (error 104) I0517 06:46:27.255704 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 19931 W0517 06:46:27.283610 19746 connection.cc:512] client connection to 127.4.96.2:48633 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:48633): Connection reset by peer (error 104) W0517 06:46:27.283746 19742 connection.cc:512] server connection from 127.4.96.2:49050 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:49050): Connection reset by peer (error 104) W0517 06:46:27.283756 20068 connection.cc:512] client connection to 127.4.96.2:48633 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:48633): Connection reset by peer (error 104) W0517 06:46:27.284502 19733 connection.cc:512] client connection to 127.4.96.2:48633 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.2:48633): Connection reset by peer (error 104) I0517 06:46:27.284747 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 20056 W0517 06:46:27.319542 19746 connection.cc:512] client connection to 127.4.96.3:53311 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:53311): Connection reset by peer (error 104) I0517 06:46:27.319682 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-tserver with pid 20179 W0517 06:46:27.320358 19746 connection.cc:512] server connection from 127.4.96.3:48296 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:48296): Connection reset by peer (error 104) W0517 06:46:27.334159 19733 connection.cc:512] client connection to 127.4.96.3:53311 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.3:53311): Connection reset by peer (error 104) W0517 06:46:27.347160 19743 connection.cc:512] server connection from 127.4.96.4:35157 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:35157): Connection reset by peer (error 104) W0517 06:46:27.347239 19730 connection.cc:512] client connection to 127.4.96.4:56288 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:56288): Connection reset by peer (error 104) W0517 06:46:27.348026 19733 connection.cc:512] client connection to 127.4.96.4:56288 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.4:56288): Connection reset by peer (error 104) I0517 06:46:27.355346 4480 external_mini_cluster.cc:978] Killing /tmp/dist-test-task4QLboq/build/tsan/bin/kudu-master with pid 19734 W0517 06:46:27.379703 19733 connection.cc:512] client connection to 127.4.96.62:50667 recv error: Network error: failed to read from TLS socket (remote: 127.4.96.62:50667): Connection reset by peer (error 104) [ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/5 (6646 ms) [----------] 6 tests from EnableKudu1097AndDownTS/MoveTabletParamTest (49850 ms total) [----------] Global test environment tear-down [==========] 20 tests from 2 test cases ran. (152982 ms total) [ PASSED ] 19 tests. [ FAILED ] 1 test, listed below: [ FAILED ] AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs 1 FAILED TEST